[ipxe-devel] iPXE Prebuilt binary web interface

[Robin Smidsrød]

On 25.10.2012 19:42, Michael Brown wrote:
> On 21/10/12 02:51, xbgmsharp at gmail.com wrote:
>> Now, I am looking for the fbuid.cgi script so I can interact with it.
>> I was thinking of sending the result in JSON format, instead of
>> rom-o-mactic which is writing 'config/local/*.h' files. But to do so
>> I need to have a look at the fbuild.cgi script.
> 
> Attached is the current version of build.fcgi plus support files.  This
> is not intended for general release, and so may be awkward to set up.
> 
> This version should already support arbitrary git revisions, arbitrary
> build architectures, arbitrary DEBUG= and EMBED= parameters, and
> arbitrary config/*.h parameters.  These are all passed in via the query
> string, for maximum compatibility with user interfaces.  In particular,
> it must be possible to create a static link to build.fcgi to generate a
> particular build: this allows pasting of links in documentation, IRC, etc.

Sorry for my late delay in answering these emails, but I think you both
should put it up somewhere in a git repo (like github) so I can point
others at it if they want to try and help out. Keeping _useful_ code
hidden in private emails is not very useful. Don't be worried that it is
not useful or easy to setup right now. Let's worry about that later.

I think you've both done wonderful work in this arena. Great initiative!

I'd like to elaborate on Michael's idea that you should be able to link
to a special build using just a simple GET url, but I feel that maybe we
should try to think a bit more in depth about the architecture of how to
run a "web compile platform".

I've been looking into queuing systems lately (ZeroMQ to be specific),
and I was thinking that we could have a worker that is able to build a
specific version with specific set of parameters, and that output should
be cachable, so you don't need to redo it again if someone asks for the
same thing again. This is kind of how the NixOS packager works
(http://nixos.org/nix/). The parameters could just be a JSON or msgpack
object that contains all the required stuff to build the binary,
including the binary blobs you need to embed (like pxelinux.0 for
ipxelinux.0 builds). Then we would have a broker that the webapp can
talk to that will relay finished builds back to the webapp. Since this
could be async you could have a cmdline client that could send a build
request and a new build will be available in the webapp. In the cmdline
client you would get back a URL with a job identifier that you can point
your browser at, and then it will show you a page with an informational
page that says that your job is still processing, and once it is done it
could list all the artifacts generated (which could be more than one file).

If you want to make a webapp that is sort-of a direct downloader (like
Michael wants) then you'd fire it up when e.g. a browser links to it, it
would send away a job request, and then block until the job response is
either success or fail.

I think this should give us a really flexible architecture that could
allow us to make lots of different interfaces (including a GUI java
client people can run as a local application to talk to the compute
grid) for different platforms. It allows our compilation workers to be
spread out if we need to support more concurrency (when/if ipxe becomes
more popular). It should also allow windows user to build customized
versions of iPXE without understanding how to use Linux. The really cool
feature is that if we make a Java GUI client it can actually fetch the
PCI ID and such directly from the machine so that if people want to make
a new ROM they actually don't need to understand how to find the PCI ID
of their card.

If we want to go with this I think the first thing we need to get down
is the protocol interaction (aka the ZeroMQ topology) and the messages
that should go back and forth between the nodes. I would argue that
msgpack might be smarter to use than JSON in this case, because we want
to support sending binary data (EMBED attachments) around, and JSON is
not really the best for that.

The really cool thing with this is that we could spread out the
compilation and stuff on multiple virtual machines, and it would be able
to scale quite horizontally.

I decided to CC this to the list because of this extensive
architechtural information I wrote. I'm thinking this could potentially
be either a new GSoC project, or something some of you non-C hackers
might want to have a shot at.

What do you think?

-- Robin