Discussion:
Send stdout to a piped command
(too old to reply)
j***@mdfs.net
2020-05-02 00:27:36 UTC
Permalink
From within a program (ie, *not* in the shell) can I
start a process and pipe my standard output to it?
Ie, I want to do something like:

if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
insert magic here
}
printf("Goes to 'more' if sendout true\n");

or do I have to replace all the character output
in my program to specify a file handle?

myout=stdout;
if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
myout=fd;
}
fprintf(myout,"Goes to 'more' if sendout true\n");

I intuitively know it's possible as there are commands
that are documented as something like "if option XXX
then output is sent to more". I just can't get the
right combination of search terms to find it.

jgh
Siri Cruise
2020-05-02 02:13:50 UTC
Permalink
In article
Post by j***@mdfs.net
From within a program (ie, *not* in the shell) can I
start a process and pipe my standard output to it?
?

Working below stdio, you can close and reopen a file descriptor.

fflush(stdin); //empty stdio buffers.
fflush(stdout);
int p[2]; pipe(p); //create a pipe.
int pid = fork(); //fork.
if (pid==0) { //in child.
dup2(p[0], 0); //dup pipe read to stdin.
close(p[0]); //close unused pipe ends.
close(p[1]);
... //stdin will be the pipe read.
} else { //in parent.
dup2(p[1], 1); //dup pipe write to stdout.
close(p[0]); //close unused pipe ends.
close(p[1]);
... //stdout will be the write.
}

That should make the parent's stdout write to the pipe instead of
wherever it went before, and the child's stdin will read from the
pipe. This should work, but be cautionned stdio doesn't always
play well changing file descriptors under the FILE*. Usually the
code does an exec so stdio reinitialises with the inheritted
descriptors 0, 1, and 2.

If you want to write stdout to the original destination and to
the pipe, like tee(1), you can create a third process.

fflush(stdin);
fflush(stdout);
int p[2]; pipe(p);
int pid = fork();
if (pid==0) {
int q[2]; pipe(q);
int qid = fork();
if (qid==0) {
dup2(q[0], 0);
close(q[0]);
close(q[1]);
/*stdin will be the pipe read.*/
...
} else {
char b[8192]; int m = 0;
close(0); close(2);
close(p[1]); close(q[0]);
while ((m=read(p[0], b, sizeof b)>0) {
for (int o=0, n=m; n>0; ) {
int r = write(1, b+o, r);
if (r<=0) exit(1);
o += r; n -= r;
}
for (int o=0, n=m; n>0; ) {
int r = write(q[1], b+o, r);
if (r<=0) exit(1);
o += r; n -= r;
}
}
exit(m==0 ? 0 : 1);
}
} else {
dup2(p[1], 1);
close(p[0]);
close(p[1]);
/*stdout will be the pipe write.*/
...
}
--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
The first law of discordiamism: The more energy This post / \
to make order is nore energy made into entropy. insults Islam. Mohammed
Kaz Kylheku
2020-05-02 03:02:10 UTC
Permalink
Post by j***@mdfs.net
From within a program (ie, *not* in the shell) can I
start a process and pipe my standard output to it?
if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
insert magic here
}
printf("Goes to 'more' if sendout true\n");
Yes; it's very simple. You have a current filedescriptor
in the [1] position which is some output device.
You want that to go to the process. That means you
simply have to plant the pipe descriptor of that
process [1]:

// insert magic here

// error checking elided

// firstly, let's flush stdout of all prior output
fflush(stdout);

// now save the current stdout by duping it somewhere
int saved_stdout = dup(STDOUT_FILENO);

// now plant the pipe's stdin as stdout:
dup2(fileno(fd), STDOUT_FILENO);

// [ ... send output to pipe using stdout ...]

fflush(stdout);

// Now close the pipe:
fclose(fd);

// Pipe is still open in stdout position we nuke it
// by restoring the saved_stdout
dup2(saved_stdout, STDOUT_FILENO);

// Now close the saved_stdout temporary descriptor
close(saved_stdout);

Something like that. Good luck.

P.S. don't mix output to the process using stdout and fd.
Unless you do flushing carefully, but what's the point.
Post by j***@mdfs.net
or do I have to replace all the character output
in my program to specify a file handle?
myout=stdout;
if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
myout=fd;
}
fprintf(myout,"Goes to 'more' if sendout true\n");
Manipulations of file descriptors via dup2 and such
are documented by POSIX. Assigning to stdout isn't
documented by ISO C or POSIX, I think.
Geoff Clare
2020-05-04 12:44:33 UTC
Permalink
Post by Kaz Kylheku
Manipulations of file descriptors via dup2 and such
are documented by POSIX. Assigning to stdout isn't
documented by ISO C or POSIX, I think.
There is no requirement for stdout to be assignable (an "lvalue" in
C standard terminology), and on some systems it is not. If you try
to assign to it on such a system, you'll get a compilation failure.
--
Geoff Clare <***@gclare.org.uk>
Barry Margolin
2020-05-04 18:05:12 UTC
Permalink
Post by Geoff Clare
Post by Kaz Kylheku
Manipulations of file descriptors via dup2 and such
are documented by POSIX. Assigning to stdout isn't
documented by ISO C or POSIX, I think.
There is no requirement for stdout to be assignable (an "lvalue" in
C standard terminology), and on some systems it is not. If you try
to assign to it on such a system, you'll get a compilation failure.
You can use freopen() to redirect it to a file, though.
--
Barry Margolin, ***@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Kaz Kylheku
2020-05-04 19:48:02 UTC
Permalink
Post by Barry Margolin
Post by Geoff Clare
Post by Kaz Kylheku
Manipulations of file descriptors via dup2 and such
are documented by POSIX. Assigning to stdout isn't
documented by ISO C or POSIX, I think.
There is no requirement for stdout to be assignable (an "lvalue" in
C standard terminology), and on some systems it is not. If you try
to assign to it on such a system, you'll get a compilation failure.
You can use freopen() to redirect it to a file, though.
Unfortunately, in the context of forking a process with a pipe,
that is not useful.

There is no requirement that if we freopen the FILE *stdout stream to a
different object, that the descriptor inside that stream object will
be descrpitor 1 (STDOUT_FILENO), which is what we really need to
me manipulating if we want to redirect the output of the popen
child process.

Even if fd [1] is the lowest-numbered one that is available when the
original stream is closed, who is to say that freopen closes the
original descriptor first, and then opens the new one?

Suppose fd [0] has been previously closed; then *it* will the
lowest-numbered one that will end up used by the reopened stdout.

Then there are other problems

- lack of flexibility: the object to become stdout must be available
under a path in the filesystem. What if it's a network socket
we inherited from a parent? We have to resort to some nonportable
/dev/fd<N> trickery.

- lack of recoverability: no way to get the original stdout back.
(The original stdout could have held the last reference to the
object, which is gone forever.) We could stash it away using dup(),
but then we could do all the work at the descriptor level and
forget about freopen.

- Side effects: closing the original stdout could break a network
connection, or change the states of serial control lines.
Preventable by dup; same remarks.

freopen on a standard stream is useful only in the global initialization
of some program that is sure it will never need the original stdout
again; such an action can't be perpetrated by some random middleware
component.

To manipulate stdout/stderr/stdin redirections around fork and exec,
it's best to go through the low level descriptors, and use flushing to
make sure the stdio view of these is consistent.
--
TXR Programming Lanuage: http://nongnu.org/txr
Music DIY Mailing List: http://www.kylheku.com/diy
ADA MP-1 Mailing List: http://www.kylheku.com/mp1
James Kuyper
2020-05-04 23:35:26 UTC
Permalink
...
Post by Kaz Kylheku
Post by Barry Margolin
Post by Geoff Clare
There is no requirement for stdout to be assignable (an "lvalue" in
C standard terminology), and on some systems it is not. If you try
to assign to it on such a system, you'll get a compilation failure.
You can use freopen() to redirect it to a file, though.
Unfortunately, in the context of forking a process with a pipe,
that is not useful.
There is no requirement that if we freopen the FILE *stdout stream to a
different object, that the descriptor inside that stream object will
be descrpitor 1 (STDOUT_FILENO), which is what we really need to
me manipulating if we want to redirect the output of the popen
child process.
"The freopen() function is typically used to attach the pre-opened
streams associated with stdin, stdout, and stderr to other files." (IEEE
Std 1003.1-2017).

I'm curious - how do you reconcile that description with the possibility
that the file descriptor might be changed along with the associated file?
Post by Kaz Kylheku
Even if fd [1] is the lowest-numbered one that is available when the
original stream is closed, who is to say that freopen closes the
original descriptor first, and then opens the new one?
I think that saying that is IEEE 1003.1-2017's responsibility. It says
that freopen() first flushes the stream, closes the file descriptor
associated with stream, opens the named file, and associates that file
with that stream. The only sequencing word it uses is "first", but I
think that's sufficient to imply that all of those actions should occur
in that order.
Post by Kaz Kylheku
Suppose fd [0] has been previously closed; then *it* will the
lowest-numbered one that will end up used by the reopened stdout.
The description of close() says:

"Usage of close() on file descriptors STDIN_FILENO, STDOUT_FILENO, or
STDERR_FILENO should immediately be followed by an operation to reopen
these file descriptors. Unexpected behavior will result if any of these
file descriptors is left in a closed state (for example, an [EBADF]
error from perror()) or if an unrelated open() or similar call later in
the application accidentally allocates a file to one of these well-known
file descriptors."

Therefore, you should never have one of those three file descriptors
closed at the time freopen() is called. If the stream being reopened is
associated with one of those file descriptors, then that exact same
descriptor is guaranteed to be the lowest-numbered one available when
the open occurs.
Siri Cruise
2020-05-05 01:16:55 UTC
Permalink
Post by James Kuyper
"The freopen() function is typically used to attach the pre-opened
streams associated with stdin, stdout, and stderr to other files." (IEEE
Std 1003.1-2017).
In unix fdopen opens a file to a specified descriptor.
--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
The first law of discordiamism: The more energy This post / \
to make order is nore energy made into entropy. insults Islam. Mohammed
Kaz Kylheku
2020-05-04 20:10:31 UTC
Permalink
Post by Kaz Kylheku
Post by j***@mdfs.net
From within a program (ie, *not* in the shell) can I
start a process and pipe my standard output to it?
if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
insert magic here
}
printf("Goes to 'more' if sendout true\n");
Yes; it's very simple. You have a current filedescriptor
in the [1] position which is some output device.
You want that to go to the process. That means you
simply have to plant the pipe descriptor of that
In fact, lookee here; I did something very similar in TXR Lisp
some years ago.

The open-process and open-command functions have the property
that if you happen rebind a Lisp standard stream such as *stdout*,
*stdin* or *stderr* (to something that has a file descriptor)
then these functions will switch the underlying POSIX descriptor
around the fork/exec or popen.

So that is to say:

;; capture output of "cat < blah.txt"
;; roughly equivalent to (open-command "cat < blah.txt"):

(with-stream (*stdin* (open-file "blah.txt" "w"))
(open-command "cat"))

Of course, this can't be expected to work, and will throw
an exception:

;; cannot cat string stream

(with-stream (*stdin* (make-string-input-stream "abc"))
(open-command "cat"))

An in-memory string stream doesn't have a POSIX descriptor.

How does that work in the working case?

It's done via the functions fds_swizzle and fds_restore.

The "struct save_fds" is a small structure containing three
integers, for saving stdin, stdout and stderr:

val open_command(val path, val mode_str)
{
struct stdio_mode m, m_r = stdio_mode_init_r;
val mode = normalize_mode_no_bin(&m, mode_str, m_r);
int input = m.read != 0;
struct save_fds sfds;
FILE *f = 0;

fds_init(&sfds);

uw_simple_catch_begin;

fds_swizzle(&sfds, (input ? FDS_IN : FDS_OUT) | FDS_ERR);

f = w_popen(c_str(path), c_str(mode));

if (!f) {
int eno = errno;
uw_throwf(errno_to_file_error(eno), lit("error opening pipe ~s: ~d/~s"),
path, num(eno), string_utf8(strerror(eno)), nao);
}

uw_unwind {
fds_restore(&sfds);
}

uw_catch_end;

return set_mode_props(m, make_pipe_stream(f, path));
}

The fds_swizzle function takes a mask which tells it which of the three
descriptors to swizzle. For instance if FDS_OUT is included, then it
will examine the current *stdout* Lisp stream, and if there is a file
descriptor there, it will plant it as descriptor 1, saving the existing
one via dup into the fd_save structure.

Then fds_restore will do the dup and close. Here are those two:

static void fds_swizzle(struct save_fds *fds, int flags)
{
if ((flags & FDS_IN) != 0)
fds->in = fds_subst(std_input, STDIN_FILENO);

if ((flags & FDS_OUT) != 0)
fds->out = fds_subst(std_output, STDOUT_FILENO);

if ((flags & FDS_ERR) != 0)
fds->err = fds_subst(std_error, STDERR_FILENO);
}

static void fds_restore(struct save_fds *fds)
{
if (fds->in != -1) {
dup2(fds->in, STDIN_FILENO);
close(fds->in);
}

if (fds->out != -1) {
dup2(fds->out, STDOUT_FILENO);
close(fds->out);
}

if (fds->err != -1) {
dup2(fds->err, STDERR_FILENO);
close(fds->err);
}
}

The swizzing is done via fds_subst.

static int fds_subst(val stream, int fd_std)
{
int fd_orig = c_num(stream_fd(stream));

if (fd_orig == fd_std)
return -1;

{
int fd_dup = dup(fd_std);

if (fd_dup != -1) {
dup2(fd_orig, fd_std);
return fd_dup;
}

uw_throwf(file_error_s, lit("failed to duplicate file descriptor: ~d/~s"),
num(errno), string_utf8(strerror(errno)), nao);
}
}

This calls stream_fd, which will throw an exception if the stream doesn't
support redirection. Otherwise we get a Lisp number out that we convert
to a C number and proceed. If this is already the descriptor we want,
e.g. *stdout* is the original *stdout* with descriptor 1, we need not do
anything, otherwise we do the dup/dup2 dance.
m***@gmail.com
2020-09-21 11:42:14 UTC
Permalink
Post by j***@mdfs.net
From within a program (ie, *not* in the shell) can I
start a process and pipe my standard output to it?
if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
insert magic here
}
printf("Goes to 'more' if sendout true\n");
or do I have to replace all the character output
in my program to specify a file handle?
myout=stdout;
if (sendout) {
fd=popen("more","w");
if (fd==NULL) { /* couldn't run etc. */ }
myout=fd;
}
fprintf(myout,"Goes to 'more' if sendout true\n");
I intuitively know it's possible as there are commands
that are documented as something like "if option XXX
then output is sent to more". I just can't get the
right combination of search terms to find it.
jgh
Loading...