Printable version of this document
Frequently Asked Questions
Contents
Getting Started
-
Who is Eligible to use Emulab.Net?
In principle, almost any research or educational use
by those that have a need for it is appropriate and encouraged.
This includes use by universities, industrial research labs, and both
US and non-US institutions. With some provisos, use for development
and evaluation is also acceptable, even by companies.
See our posted policies
for more detail. If you are unsure about your eligibility to use
Netbed/Emulab, please just send us an email inquiry.
-
How do I start a project?
If you are new to the Testbed, simply click on the "Start Project"
link on the Emulab Home
Page. You will need to fill in the forms with your personal
information and information about the project. Then click on the
"Submit" button. Within a few days you will be contacted via email
with an approval message. More information about starting projects
can be found in
Authorization Page.
If you already have an Emulab account, and wish to start a second
project, first log into the Web Interface. Then select the "Start
Project" link; all of the personal information will already be
filled in. You will need to complete just the project information
section.
-
How do I join a project?
If you are new to the Testbed, simply click on the "Join Project"
link on the Emulab Home
Page. You will need to fill in the form with your personal
information, and provide the name of the project you are trying to
join (typically, the Project Leader will have told you the
name of the project). Then click on the "Submit" button, and wait
for an email with your new user key. When that email arrives, use
the link in it (or the key itself), and use it with your password
to log into the web site and verify your account. Then just wait
for the project leader to approve you. When approved you will
receive an email message saying so, and you can then log into the
Testbed.
-
I'm a project leader,
and someone applied to join my project,
but they're not on the list to be approved.
Joining a project has 3 stages. The first two are done by
the person trying to join, and they both must be completed
before you can approve their application. The first two are
outlined in the previous question, where the
user fills out the "Join Project" form, and performs account
verification. After these two steps are both complete, the
project leader and any group leaders in the group
(More info here) will get an email saying
the account is ready to be approved, and it will appear on the
list of new users waiting to be approved.
If someone says they've applied, but you haven't received an
email from Emulab about it, and they don't show up on your
list, the most likely cause is that they haven't finished the
verification step.
-
Will Emulab send me email messages?
Yes! Emulab uses email notifications to you in several different
ways. Often it will send you a copy of information regarding
experiments you set up, applications to projects, and other
things you do at Emulab. Sometimes (like with account
verification) the email is a critical part of being able to use
Emulab.
For those reasons it is critical that any spam filtering
software you have accept email from Emulab itself (anything
coming from the emulab.net domain) and from Emulab staff (from
the cs.utah.edu or flux.utah.edu domains). In many cases, it may
also say that is from a specific machine in those domains. Our
messages usually do not get flagged as spam by most filters, but
in certain cases it can be a problem. It is also important that
it not require manual intervention or confirmation to get emails
through to you, so programs like SpamKiller can cause problems.
It is also important to read your email often while you are
using Emulab, especially while you have machines reserved in
an experiment. A few emails may be the only notification you'll
get before we swap out an experiment that appears to be idle,
and if you don't respond, you may lose important work. Email is
also our method for informing you about problems we may be
experiencing, downtimes, or other important announcements. Your
experience with Emulab (and ours with you) will be much more
pleasant if everyone is responsive to email.
-
I have an Emulab account. Now what?
Once you have been approved to start (or join) your first project,
you will be able to log into Emulab's user machine,
users.emulab.net. We require that all Emulab users use ssh. For
example, if your Emulab account name is "joe", then you would do:
ssh users.emulab.net -l joe
Your password is the same as the password you supplied to the
Start (or Join) Project web page.
-
Do I need any
special directories in my PATH variable?
There are several useful (although not required) programs installed on
users.emulab.net in /usr/testbed/bin. You should edit
your dot files to include that directory in your search path.
-
Can I be in more than one project?
Yes. You may join (and/or start) as many projects as you like,
subject to Emulab
administrative policies.
-
Can I change my Emulab password?
Yes. You can change your Emulab Web password and your Emulab login
password (the password you use to log into users.emulab.net, as
well as nodes in your experiments). To change your password,
simply click on the "Update User Information" in the menu to your
left, and then enter your new password in the location provided.
Your new password will be installed on users.emulab.net
immediately. Your experimental nodes will get the new password
when they reboot.
-
I'm a project leader. Can I designate TAs?
Yes. To designate a TA, you must first create a project
group. A project group is a lot like a unix group, and
in fact unix groups is the mechanism used to protect members of
one group from members of another group. When you create a group,
you designate a group leader who is responsible for
approving users who apply to join the group. Group leaders may
also terminate experiments that have been created by members of
the group. As Project Leader, you may also shift members of your
project in and out of your project's groups as you like, and you
are automatically a member of all groups within your project. As a
convenience, all new projects are created with one new group,
termed the default group. As its name implies, whenever
the group is left unspecified in a form, it defaults to the
project group (this allows you to create a project without any sub
groups at all; new members join the default group, new experiments
are created in the default group, etc.).
Project groups are created via the Project Information link at
your left. Simply go to the project page in which you want to
create a group, and look for the "Create New Group" link. More
information on project groups is available via the Emulab Documentation page in the
Groups Tutorial.
-
How do I report a specific problem?
Don't hesitate to send us email!
Ok, hesitate just a little and read the rest of this entry first.
Before sending email, be sure to check out the
Troubleshooting
entry which describes several common problems and possible causes.
If you do send email, there are several pieces of information you
should include to make our job easier.
- The affected project and experiment name is pretty obvious,
but some people forget.
- The particular nodes, OSes, and programs
involved will help us zero in on the problem more quickly.
- The time at which the problem occurred.
We need this to correlate with our various log files.
- Any relevant observations and actions you have attempted.
Did the problem only happen once? Is it reproducible? Did restarting
a program or rebooting a node help?
After sending email,
please do not swap or terminate the experiment, if possible.
It can be a lot harder to track down a problem after the experiment
is gone. Sometime you may need to terminate it, for example it is
in the middle of the night our time and you really need the nodes
that are tied up in the experiment. But at least give us 15 minutes
to respond to your message before acting, and then let us know that
you did swap/terminate it.
-
Where do I get more help?
If you cannot find an answer to your question in the
Emulab Documentation, then you can
send us an email message. We will try
to answer your question as quickly as we can.
Using the Testbed
-
Is there a tutorial?
Yes, we have an extensive tutorial
on using the Testbed.
-
Do you have a GUI to
help me create experiments?
Yes, we provide a GUI that gives you an easy to use drawing
palette on which you can place nodes, lans, and links. Testbed
specific attributes such as operating system, hardware type, and
link/lan characteristics, may be attached to each object. With a
single click, you can instantiate your new topology on the Testbed
as an experiment in one of your projects. Alternatively, you can
save the auto-generated NS file on your machine, edit as required,
and then submit it later when creating an experiment.
To access the GUI, please log in and go to the Begin Experiment page.
Note: you need a Java compliant browser.
-
Are there any constraints on my topology?
Yes, but only those imposed by the physical hardware that is
currently available in our testbed. The constraints that people
most commonly run into are the maximum speed of our links
(100Mbps) and the maximum number of network interface cards (NICs)
in our machines. You can't get any links faster than 100Mbps,
since we don't yet have gigabit links for experimenters to
use. Our nodes each have 4 experimental network interfaces, so
each node can be a member of up to 4 links or LANs. A good
strategy for making your topology fit within those limits is to
replace multiple links to a node with a LAN or with a router
node.
Another approach is to use
emulated network links. Emulated links are multiplexed
over the actual physical links, up to the physical bandwidth that
the physical link can support. In other words, five 20Mb links can
be multiplexed over a 100Mb physical link. More information on
emulated links can be found here. Ask
Testbed Operations if you need further assistance.
-
How many nodes can I ask for?
You can ask for as many nodes as are currently available! You can
click on the "Node Reservation Status" link at your left to see
how many nodes are currently free. If you ask for more than are
currently available, your experiment will be rejected (you
will receive email notification shortly after you submit your NS
file to the web interface).
We urge all new Emulab users to begin with a small 3-4 node
experiment so that you will become familiar with NS syntax and the
practical aspects of Emulab operation.
-
How long can I keep using my nodes?
You can keep them as long as you need them, subject to our Node Usage
Policies. In general, you should do your work, and then
terminate your experiment as soon as you're done with it. If
you're not done with it, but are through for a while, you should
probably "swap out" your experiment (See the question What is Swapping in this FAQ). It is
especially important to swap out your experiment if you're through
with it for the weekend. Emulab usually gets heavy use on the
weekends by users who need to make very large experiments, so it
is important to leave as many nodes available as possible.
-
I only need a couple of nodes but none are free,
should I just keep trying?
It is better for you and us if you don't just keep hitting the
submit button every few minutes. It wastes your time and floods
us with email (we get every failure message you do!)
Instead, you can now use the
Batch System
to queue an interactive job. By submitting your experiment as
a batch job, but without any
tb-set-node-startcmd
directives in your NS file, the job will be queued until nodes are
available. For most experiments, this means just using your regular
NS file, and checking the Batch Mode Experiment box when you create
the experiment.
When your queued job is swapped in, you will be sent email to
inform you, and you can start working!
Please note that the
experiment will be idle when it is swapped in, and will be
idle swapped if you do not
get things running on the nodes in a short period of time. If your
experiment does get swapped out before you can get to it, you can
always visit the experiment's information page and try again by
using the Queue Batch Experiment menu item.
-
What if I need more nodes than are free?
For example, say you need 50 nodes but there are only 40 free. In
general, getting this many nodes is going to require intervention
from Testbed Operations, if only so we can ask other experimenters
to free up nodes, if possible. Please send us email if you are not
able to able to get the number of nodes you need for your experiment.
Another alternative is to use the
Batch System.
If your experiment is amenable to being batched (does not require
human intervention to start and stop), then you can submit a batch
request, which will be serviced when enough nodes become
available. Typically, you would start out with a few nodes,
getting used to the batch system and creating whatever scripts are
needed to make the experiment batchable. Then scale up to larger
numbers of nodes. Thats the easiest way of getting a lot of nodes!
-
Do I get root access on my nodes?
Yes. Project leaders get root access to all of the nodes in all of
the experiments that are running in their project. Project members
get root if their project leader grants them root access, when the
leader approves the group membership request.
Root privileges are granted via the sudo command. The
tutorial describes
this in more detail.
-
Do my nodes have consoles I can look at?
Yes. Each of the PCs has its own serial console line with which you can
interact, either directly from your desktop (see next FAQ entry),
or by hopping through the "users" machine,
using our console program. To connect over serial line to
"pc1" in your experiment, ssh into users.emulab.net, and
then type console pc1 at the Unix prompt. You may then
interact with the serial console (hit "enter" to elicit output from
the target machine).
In any case, all console output from each node is saved
so that you may look at it it later. For each node,
the console log is stored as /var/log/tiplogs/pcXXX.run.
This run file is created when nodes are first allocated
to an experiment, and the Unix permissions of the run files permit
only members of the project to view them. When the nodes are
deallocated, the run files are cleared, so if you want to save
them, you must do so before terminating the experiment.
-
How do I connect directly to node consoles,
without going through users?
Clicking "Connect to Serial Line"
in the Node Options page will send your browser a "text/x-testbed-acl"
".tbacl" file.
In windows, if you have installed tiptunnel, available
below, you can save this file in a folder and double-click it
to launch a tunneled connection to your node.
In FreeBSD or Linux, you can save the file and pass it as an argument
to tiptunnel, or associate it with
tiptunnel in your web browser.
Upon connection you typically first have to hit "enter" to
elicit output from the target machine.
Instructions for Windows:
- Run the installer executable, and successfully complete the installation.
- In the Web Interface Node view,
click on the "Connect to serial line" link.
- Save the resulting .tbacl file in an appropriate place.
(for instance a folder off the desktop.)
- For the lifetime of your experiment, you can simply double-click
these .tbacl files to connect.
Instructions for Linux/FreeBSD:
- Use
gunzip,
then tar xvf on the downloaded file.
- Move the resulting
tiptunnel binary into
a directory of your choice (/usr/local/bin,
or ~/bin are two good places.)
- Set up your browser to handle MIME type "text/x-testbed-acl"
as outlined below.
- In the Web Interface Node view,
click on the "Connect to serial line" link.
- If your browser is properly configured to use
tiptunnel,
a new xterm window with a telnet session open to your node
should emerge.
- (Alternately, you can tell your browser to save "text/x-testbed-acl"
files in a directory and you can run them with
tiptunnel directly;
this may be more convenient than using the
web interface every time you wish
to connect to a node in your experiment.
Note that these files are valid for the
lifetime of your experiment.)
Linux/FreeBSD and Netscape 4.7:
- Choose
preferences from the edit menu.
- Select
Navigator, then Applications under
it.
- Click the
New... button.
- In the
MIMEType box, type text/x-testbed-acl
- In the
Suffixes box, type tbacl
- Choose
Application in the Handled by
box
- Next to
Application, either type the path to the
tiptunnel binary, or use Choose... to find
it.
- Now, be sure to put a space, then
%s after the
path to the application in the box. This tells netscape to actually
pass the aclfile into tiptunnel (Mozilla does not require this;
see below.)
- Click
OK, then OK again.
- Clicking a "connect to serial line" link should now
bring up a connection in an xterm window.
Linux/FreeBSD and Mozilla:
- Choose
preferences from the edit menu.
- Select
Navigator, then Helper Applications
under it.
- Click the
New Type... button.
- In the
MIMEType box,
type text/x-testbed-acl
- In the
File extension box, type tbacl
- For
Application to use, either type the path to the
tiptunnel binary, or use Choose... to find
it.
- In Mozilla do not add a
%s.
- Click
OK, then OK again.
- Clicking a "connect to serial line" link should now
bring up a connection in an xterm window.
-
Can I reboot (power cycle) my nodes?
Yes. Each of the PCs is independently power controlled. If your
node hangs, or is otherwise unresponsive, you can use the
node_reboot command, as discussed in the
Emulab Tutorial.
-
I've clobbered my disk! Now what?
If you manage to corrupt a disk (or slice), no worries. You can easily
repair the damage yourself by reloading a fresh copy of the default
disk image. You will of course lose anything you have stored on that
disk; it is a good idea to store only data that can be easily recreated,
or else store it in your project directory in /proj. Disk
reloading is covered in more detail in the
Emulab Tutorial.
-
Where do I
store files needed by my experiment?
Each project has its own directory, rooted at /proj,
which is available via NFS to all of the nodes in experiments
running in that project. For example, when the "RON" project was
created, a directory called /proj/RON was also created. This
directory is owned by the project creator, and is in the unix
group "RON." Its permission (mode) is 770; read/write/execute
permitted by the project creator and by all of the members of the
project RON, but protected against all access by people outside
the RON project.
Sub-groups within a project likewise have a directory in the
/groups tree. A group named "group1" in the RON project
would thus have a group directory in /groups/RON/group1.
Project members are encouraged to store any files needed by their
experiments in the corresponding /proj or /groups directory.
-
Are my files on
users.emulab.net backed up (filesaved)?
Yes. All of the files in your home directory on /users, all of
the files in your project directory in /proj, and all of the files
in your groups directory in /groups are filesaved. While
we can restore lost files in an emergency, we encourage you to
back up critical data on your own to avoid (possibly long) delays
in conducting your experiments.
-
Are the nodes
in my experiment backed up (filesaved)?
No! The nodes in your experiment are not filesaved. Any changes
you make to the local filesystems will be lost if the event of a
disk failure. We plan to provide a mechanism for experimenters to
create snapshots of their node state, but that is not done yet. In
the meantime, any files that must not be lost should be stored in
the project directory (/proj/), which is available
via NFS to all of the nodes in your experiment. You may also store
files in your home directory (/users/), also available via
NFS to all of your nodes, but that is not the preferred location
since quotas on /users are relatively small.
-
What is Swapping?
Swapping is when you (or we, or the Emulab system) temporarily swaps
out your experiment,
releasing all of the nodes in the experiment. Your experiment is
still resident in the Emulab database, and you can see its status
in the web interface, but no nodes are allocated. Once an
experiment is swapped out, you can swap it back in via the web
interface by going to the Experiment Information page for your
experiment, and clicking on the swapin option.
You can also modify it.
The idle-swap checkbox in the Begin Experiment web page
is used to determine what experiments can be automatically
swapped by the testbed scheduling system. Note that all experiments are
capable of being swapped; even if you do not check the idle-swap box,
you are free to swap your own experiments as you like. The only
difference is that the testbed scheduling system will not consider your
experiment when looking for experiments to swap out. You will sometimes
notice that the Experiment Information page does not contain the swap
link. That is because experiments cannot be swapped when they are in
transition. For example, when the experiment is being swapped in (say,
after first being created) the link will disappear until the experiment
is fully swapped in, and it is capable of being swapped out. You will
need to occasionally reload the page so that the updated state is
recognized and the swap link appears.
Be aware that we do not currently save any files that you may have
placed on your nodes. When your experiment is swapped back in, you
will likely get different nodes, with fresh copies of the disk
images. For that reason, you should not swap your experiment out
unless you make arrangements to save and restore any state you need.
Please be sure to read our
Node Usage Policies,
which contain detailed information on swapping.
-
What is Experiment Restart?
Experiment restart (or perhaps more aptly, replay) allows you to
rerun your experiment from scratch, but without the added expense
of a swapin and swapout. In other words, the nodes that are
currently allocated to your experiment are all rebooted, and the
experiment startup state is cleared.
The event scheduler for the experiment is restarted, and your event
sequence is replayed again. Note that your rpms and tarfiles are
not installed again. Replay is obviously faster than
swapout/swapin, and has the added benefit that you will not run
the risk of not being able to swapin for lack of available nodes.
-
How can I
get switch statistics (such as packet counts) for my experiment?
We have a command called portstats that allows you access
to some of the port counters on our switches. To use it, you'll need
to ssh to users.emulab.net. 'portstats <proj>
<exp>' will get you stats for all experimental interfaces in
your experiment. Run 'portstats -h' to get a list of other
options, such as different sets of stats.
Note that the numbers returned by portstats do not get
reset between experiments.
-
What names should I use to refer to the nodes in my
experiment?
We set up names for your nodes in DNS, for use from outside,
and /etc/hosts files for use on the nodes in the experiment.
Since our nodes have multiple interfaces (the control network, and,
depending on the experiment, possibly several experimental interfaces,)
determining which name refers to which interface can be somewhat
confusing. The rules below should help you figure this out.
- From the outside world -
We set up names in the form
node.expt.proj.emulab.net in DNS,
so that they visible anywhere on the Internet. This name always refers
to the node's control network interface, which is the only one
reachable from the outside world.
- On the nodes themselves -
There are three basic ways to refer to the interfaces of a node. The
first is stored in DNS, and the second two are stored on the node in
the
/etc/hosts file.
- Fully-qualified hostnames - These names the same ones
visible from the outside world, and referred to by attaching the
full domain name: ie.
node.expt.proj.emulab.net. (note
that, since we put .emulab.net in nodes' domain
search paths, you can use
node.expt.proj as a shorthand.)
This name always refers to the control network
node-link form - You can refer to an
individual experimental interface by suffixing it with the name of
the link or LAN (as defined in your NS file) that it is a member
of. For example, nodeA-link0 or
server-serverLAN. This is the preferred way to refer
to experimental interfaces, since it uniquely and unambiguously
identifies an interface.
- Short form - If a node is directly connected to the
node you're on, you can refer to that node simply with its name
(eg.
nodeA.) Note that this differs from the fully-
qualified name in that no domain is given. We also create short
names for nodes you are not directly connected to.
However, if two nodes are
connected with more than one interface, or there is more than
one route between them, there is no guarantee that the short name
has been associated with the one is on the 'best' (ie. shortest or
highest bandwidth) path - so, if there is ambiguity, we strongly
suggest that you use the node-link form.
-
Can I modify my experiment after creating it?
Yes. On the experiment view page, choose "Modify this Experiment".
This will allow you to modify an experiment, either swapped-out or in,
by editing its NS file.
If the experiment is swapped-out,
Experiment Modify will simply replace its topology with the
newly specified one;
this new topology will be mapped when the experiment is swapped in.
If the experiment is already swapped-in, Modify will change the topology
and map in the portions which have been changed. This allows dynamic
addition, subtraction, and replacement of an experiment's nodes and links.
However, when modifying swapped-in experiments, there are a couple
things to keep in mind:
- Any node with the same name in the old and new topology will remain
on the same physical machine, unaffected-- its disk will not be reloaded.
If you want to, for example, change the hardware on a machine, you will
have to call the machine something different in the new topology.
-
It is highly recommended that you leave the
"Reboot nodes in experiment" box checked in the Experiment Modify form.
This is especially important if changing your experiment topology
(adding or removing nodes, links, and LANs).
If adding/removing a delay to/from an existing link, or replacing
a lost node without modifying the experiment topology,
this won't be necessary.
-
The event system is not automatically restarted for your
experiment, so you will not be able to modify the traffic
shaping for new links. In addition, if you add program
agents or traffic generators, these will not activate unless
you restart the event system by hand. Unfortunately, all
events will be replayed, so be careful: On
users.emulab.net, you may do the following:
eventsys_control [-f] replay proj expt
-
Are there Linux and/or FreeBSD sources and packages available locally?
Yes. We provide sources and packages for a few versions of
FreeBSD and RedHat Linux. The place to look for available
software is under /share on either users.emulab.net
or your experimental nodes. This path is readonly (and NFS
mounted on the nodes), so you'll need to make a copy of the
contents found there if you need to do more than reference them
for information or installation.
- FreeBSD Paths:
The FreeBSD kernel, and userland sources are available under
/share/freebsd. Look there to see if the version you are
seeking is available. You can also find Emulab additions and
modifications to FreeBSD here. Emulab kernel configurations are
called TESTBED and located in sys/i386/conf relative to the
FreeBSD source trees. The README file in this directory has more
information on the contents.
- Linux Paths:
Linux kernel sources and RPMs for various versions of Redhat can
be found under /share/redhat. Look there to see if the
version you are seeking is available. You can also find Emulab
additions and modifications to Linux here. Emulab kernel
configurations are called config-emulab and exist in the
root of the kernel source trees. The README file in this
directory has more information on the contents.
- Other Software:
We provide a few other generally useful software packages and
sources under /share as well. Have a look around.
- Something Missing?
If you think something should be added to /share, feel
free to send your suggestion(s) to us via
email. Note that we may retire some
offerings if we determine them to be of little value.
Hardware Setup
-
What kind of computers are used for my nodes?
-
How many nodes are there?
-
How many ethernet cards are on each node?
Please see the
Hardware Overview page for a description and count of the
computers that comprise the Testbed.
-
How do I ask for specific hardware?
See the previous FAQ entry for a description of what hardware is available.
If you would like to ask for a specific type of hardware, such as a
pc850, see the
tb-set-hardware
command in our NS
extensions document. You can also define classes of nodes which should
be given the same hardware using
virtual
type commands.
-
How many nodes are currently available (free)?
If you click on the "Node Reservation Status" link in the menu to
your left, you will see a summary of the number of nodes (by type)
that are currently available, followed by a listing of the
reservation status of each individual node.
-
Can I do traffic shaping on my links?
Yes! You can specify the delay, bandwidth, and packet loss rate
between any two nodes in your topology. Bandwidth and delay are
specified in the NS duplex-link statement, while packet
loss rate is specified with the Emulab tb-set-link-loss
extension to NS. You may also specify delay, bandwidth, and packet
loss rate between nodes in a regular LAN.
Please see the
Extensions
page for a summary of all Emulab NS extensions, and the
Emulab Tutorial for an
example.
-
Can I modify the
traffic shaping parameters on my links?
Yes! If your NS file specified traffic shaping on a link, then you
can subsequently modify those parameters after the experiment has
been swapped in. Note that you cannot convert a non shaped
link into a shaped link; you can only modify the traffic shaping
parameters of a link that is already being shaped. To modify the
parameters, go to the Experiment Information page of your
experiment, and click on the "Control Traffic Shaping" menu
option. Follow the instructions at the top of the page.
An alternative method is to log into users.emulab.net and use the
delay_config program. This program requires that you know
the symbolic names of the individual links. This information is
available via the web interface on the Experiment Information
page. The command line syntax for delay_config will be
displayed when the -h option is given.
-
Are there other
traffic shaping parameters besides latency,
bandwidth, and packet loss rate?
Yes! Please see the
advanced tutorial. Note though, that these other parameters
can be specified for duplex links only (not lans), and that they
are not configurable with delay_config, but with a
different testbed utility call tevc (also described in
the advanced tutorial).
Software Setup
-
What OS do the nodes run?
Please see the
Software Overview page for a description of the Operating
Systems that can be run on each of the Testbed nodes.
-
How do I select which OS to run on each node?
When a choice of OS is available, you may specify which one you
prefer for each node in the NS file using the Emulab
tb-set-node-os extension to NS. When your experiment is
configured, the appropriate disk image will be loaded on your
nodes, and the selected operating system will boot up on each.
Please see the
Extensions
page for a summary of all Emulab NS extensions, and the
Emulab Tutorial for an
example.
-
Can I load my own software (RPMs/Tarballs) on
my nodes?
Yes! If you have an RPM or Tarball (or more than one) that is appropriate for
loading on the OS you have selected, you can arrange to have them
loaded automatically when your experiment is configured. The
Emulab NS extension, tb-set-node-rpms, is used in the NS
file to specify a list of RPMS to install.
The tb-set-node-tarfiles
command is used to specify a list of Tar
files to install, as well as the directories in which the tars
should be unpacked. You may specify a
different list for each node in the experiment. The RPMs/Tarballs
will be installed when the nodes first boot after the experiment is
swapped in or if the nodes detect a changed RPM/Tarball during a reboot.
Please see the
Extensions
page for a summary of all Emulab NS extensions, and the
Emulab Tutorial for an
example.
-
Can I schedule
programs to run automatically when a node boots?
Yes! You can arrange to run a single program or script when your
node boots. The script is run as the UID of the experiment
creator, and is run after all other node configuration (including
RPM installation) has completed. The exit status of the script (or
program) is reported back and is made available for you to view in
Experiment Information link in the menu at your left. The Emulab
NS extension tb-set-node-startcmd is used in the NS file
to specify the path of the script (or program) to run. You may
specify a different program for each node in the experiment.
Please see the
Extensions
page for a summary of all Emulab NS extensions, and the
Emulab Tutorial for an
example.
-
How can I turn on routing or set up routes automatically
in my nodes?
By default, we do not setup any static routes or run any routing daemon
on nodes in an experiment. However, we do provide several options for
experimenters, which are described in the
"Setting up IP routing
between nodes" section of the
Emulab Tutorial.
-
How does my software determine when other nodes in my
experiment are ready?
If your application requires synchronization amongst your nodes,
you may use the Emulab provided synchronization server, which
provides a very simple form of barrier synchronization. Use of the
synchronization server is described in more detail in the Emulab Tutorial.
-
Can I run my own Operating System?
Yes! You can run your own OS (or a customized version of an Emulab
supported OS) on any of the PCs. You can also run OSKit kernels on the
PCs. Each of the PCs is partitioned with two DOS partitions large
enough to hold the typical OS installation. The 1st and 2nd
partitions are each 3GB. The 3rd partition is 500MB, and is
labeled as Linux Swap. The 4th partition is the remainder of the
disk, and varies in size depending on the pc type. We recommend
that you use the 1st or 2nd partition; using the 4th partition
will restrict the number of machines that you can run your OS on
since it varies in size. Note that you must leave the MBR (Master
Boot Record) in sector 0 alone, and that your custom partition
must contain a proper DOS boot record in the first sector.
Please note that while users are free to customize their disks and
install their own operating systems, Emulab staff will not be able
to offer more than encouragement and advice! We cannot install the
OS for you, and we cannot load CDROMS, floppy disks, or tape drives!
We do provide an easy way for you to boot FreeBSD from a
memory based filesystem (MFS) so that you can more easily work
with the disk (in case it is not possible to install your OS on a
live disk). Beyond that, you are pretty much on your own!
Many users had great success with customizing an Emulab supported
OS (FreeBSD or Linux), and then creating a disk image that is
autoloaded when the experiment is swapped in. We strongly
encourage people to use this approach whenever possible! There is
more information available in the Custom OS section of
the Emulab Tutorial.
-
Can I share a disk image between two projects?
No. At this time you cannot share OS images between projects. We
are thinking of adding project collaboration support, but that is
a future project.
In the meantime, you will need to create an
image
descriptor in the project that wants to use your image. Fill out
the form, but leave out the "Node to Obtain Snapshot from". Then
just copy the image over to the default path it picked for you in
the form. There is more information available in the Custom OS section of
the Emulab Tutorial.
-
What if I need more disk space on my nodes?
Each node has a partition at the end of the disk that you can use if
you wish. In Linux, the partition is /dev/hda4 ; in FreeBSD,
it's /dev/ad0s4 . There is no filesystem on this partition,
so you'll need to create it yourself. Before going any further, there
is one very important point: anything you put in this disk space will
be lost when your experiment swaps out! That is, unless you create
a Custom disk image
before it gets swapped.
Recent versions of our standard FreeBSD and Linux images include a
script, /usr/testbed/bin/mkextrafs for this purpose.
Just do the following as root:
/usr/testbed/bin/mkextrafs /mnt
and it will create the filesystem, mount it on /mnt
and make an entry in /etc/fstab so that the filesystem
will be mounted on future reboots.
If that script does not exist, you can perform the steps by hand.
For example, in Linux do the following as root:
- Set the correct partition type (type 83 - ext2fs):
fdisk /dev/hda
- Press 't' to change partition sysid.
- Enter '4' as the partition to change.
- Enter '83' to specify ext2fs.
- Type 'w' to save and exit.
- Create the filesystem:
mkfs /dev/hda4
- Mount the filesystem:
mount /dev/hda4 /mnt
You may want to add the filesystem to /etc/fstab
so that it will be automatically mounted on future reboots.
In FreeBSD, do the following as root:
- First, you need to set the correct partition type:
fdisk -i4 /dev/ad0
- Do not change what the BIOS thinks.
- Edit the partition info, setting the sysid to 165; leave other
metrics alone.
- Do not change the active partition
- Write out the new partition table
- Next, you have to create a BSD disklabel on the partition:
disklabel -w ad0s4 auto
- Create the filesystem:
newfs /dev/ad0s4c
- Finally, mount it:
mount /dev/ad0s4c /some/where
You may want to add the filesystem to /etc/fstab
so that it will be automatically mounted on future reboots.
The available space ranges from 6-33GB depending on the disk type.
-
Are there testbed-specific
daemons that could interfere with my experiment?
Probably not.
By default, the testbed startup scripts currently start two daemons in
addition to the OS's standard set. Other daemons may be started depending
on the network services you ask for in your ns file (see below).
Unconditionally started daemons:
healthd - A low overhead hardware health monitor.
This deamon periodically polls the machine's health monitoring
hardware and sends this information back to our boss
node for analysis. The hardware is polled once per second, and a
status datagram is sent out once every five minutes.
Healthd's overhead is quite low, but it can be safely
killed and disabled from startup if you're worried about possible
side effects. It is started by
/etc/testbed/rc.healthd.
slothd - A low overhead usage analysis tool.
Slothd is important to efficient testbed utilization
and should run on every node whenever possible. Its overhead is
almost negligible (essentially less than running 'ls -l
/dev' once every five minutes), and should not interfere with your
work. However if your experiment is exceptionally sensitive, then
you may arrange with us to disable slothd. Please
note that we will restart this daemon if it is not running unless
prior arrangements have been made.
Conditionally started daemons:
gated - A network routing daemon.
If you have requested automatic routing on your nodes with
$ns rtproto Session in your NS file, this will
start gated on all of your nodes.
We have left all daemons started by the operating systems' default
configurations (such as cron) enabled, so you should also
look at them if you are concered about running processes affecting
your experiment.
-
Does Emulab support IP Multicast?
In short, yes, the local nodes in Emulab (but not all remote
Netbed nodes) support IP Multicast on the experimental
network. In order to use it, you must have a kernel that
supports it, and if you want multicast routing, you'll need to
enable mrouted. (You can do it manually, or
automatically via program objects or startup commands, but the
rtproto commands will not do it.)
When using multicast, there are a few issues you need to be aware
of. The first is the fact that multicast traffic will often find
the control network, rather than the experimental network, which
you don't want. See this
section of the tutorial for information about the control net.
There are two ways to work around the control net. The first is to
set a route for all multicast addresses (224.0.0.0/4) to go out the
experimental interface of your node. The second is to have your program
use the IP_MULTICAST_IF sockopt to bind to a particular interface.
You should also stay away from multicast addresses that have special
meanings, such as 224.0.0.1 . You can get a list of these addresses
from IANA
here.
Security Issues
-
Is Emulab Firewalled?
Yes. Emulab blocks all of the low numbered ports (ports below 1024),
with the exception of ports 20 and 21 (FTP), 22 (Secure Shell), and 80
(HTTP). This is for the protection of experimenters, as well as to ensure
that an errant application cannot become the source of a Denial of Service
attack to sites outside of Emulab. If your application requires external
access to other low numbered ports, please contact us to make special
arrangements.
Troubleshooting
-
My experiment setup failed,
what did I do wrong?
Experiments can fail in many, many ways, but before you send the above
vague question off to us, consider a couple of things.
First, look carefully at the "experiment failed"
e-mail that you received. It includes a log of the setup process which,
while not a model of clarity, often contains an obvious indication of what
happened.
One potential point of failure is the mapping phase where Emulab
attempts to map your topology to the available resources. Look
in the log for where it runs assign. Common errors
here include:
If the setup log shows assign failing repeatedly and
eventually giving up, contact us.
The next potential failure point is the setup of the physical
nodes. If you are explicitly setting the OS image to use with
tb-set-node-os, then make sure you have specified
a valid image (e.g., did you spell the OS identifier correctly?)
Again, the log output should include an error if the OSID was
invalid. Try:
os_load -l
on users.emulab.net to get a list of OSIDs that you can use.
If the OSID is correct, but the log contains messages of the form:
*** Giving up on pcXXX - it's been NN minute(s).
*** WARNING: pcXXX may be down.
This has been reported to testbed-ops.
then a node failed to reach the point where it would report a successful
setup to Emulab.
Near the end of the experiment setup, Emulab's event system can fail to
startup with a message like this:
Starting the event system.
*** ~/.ssh/identity is not a passphrase-less key
You will need to regenerate the key manually
*** /usr/testbed/devel/stack/sbin/eventsys.proxy:
Failed to start event system for foo/bar
Or, like this:
Starting the event system.
Permission denied, please try again.
Permission denied, please try again.
Permission denied.
*** /usr/testbed/devel/stack/sbin/eventsys.proxy:
Failed to start event system for foo/bar
This failure occurs because you have manually changed your default SSH
identity (~/.ssh/identity) or edited your authorized_keys file in your
Emulab home directory without going through the "Edit SSH Keys" web form on
your user page. The easiest way to fix this is to make sure the passphrase
is empty using ssh-keygen(1) on the user's machine:
users$ ssh-keygen -p -P "<old-passphrase>" -N "" -f ~/.ssh/identity
Then, make sure the corresponding public key in your Emulab home directory
("~/.ssh/identity.pub") is listed in the "Edit SSH Keys" form.
Such failures can be caused by many things. Sometimes
a transient load on an Emulab server can push a node over its
timeout, though this is happening less and less as we
improve our infrastructure. Most often, these failures are caused
by the use of custom images which either do not boot or do not
self-configure properly. These are harder to dianose because you
often need access to the console logs to see what happened,
and these logs aren't available after an experiment fails.
However, it is possible to interactively monitor
the console while the experiment is setting up since console access
is granted early in setup process. You can either use the
console command on users, use the
tiptunnel client application,
or just run "tail -f" on the /var/log/tiplogs/pcXXX.run
file.
-
My experiment is set up, but I cannot
send packets between some of the nodes. Why?
The most common reason is that your topology
includes nodes which are not directly connected, and you have
not setup any routing. Refer to
"How can I turn on routing or set up routes
automatically in my nodes?" for details. If you cannot
send packets between two machines which are directly connected
(via a link or a lan), then there are two possibilities:
either the nodes did not properly negotiate their speed and
duplex with the Cisco switch, or the physical wire is loose
or bad. In these cases, you should contact us
for help.
-
I asked for traffic shaping,
but everything seems to be going at full LAN speeds.
What's wrong?
The most likely problem is that it is using the unshaped control
network for the traffic you're looking at. This occurs when it
tries to contact a node using a "pcXXX" address, like pc76 or
pc76.emulab.net, or when it tries to ping a fully-qualified name,
like NodeA.myexpt.myproj.emulab.net, which also resolves to a
control network address. On one of your nodes, take a look at the
file /etc/hosts. It shows the IP addresses and aliases that refer
to the different experimental interfaces. These are the names/IPs
you can use to see the delays.
See this section
of the tutorial for more details on the control network.
For a discussion of the way to 'name' interfaces on the control
and experimental networks, see the the naming
section of this document.
-
I set a non-zero packet-loss (or delay) but 'ping'
shows no packet-loss (or delay). Why?
You are probably pinging through the control net interface. See this
Troubleshooting FAQ entry and
the control net section
of the tutorial.
-
I set a non-zero packet-loss (or delay) but 'ping'
shows a different packet-loss (or delay). Why?
Short answer: Ping is round trip, PLR and delay are "one way".
Long Answer: If you're not seeing any traffic shaping at all
(100Mbps, 0ms, 0plr), see this FAQ entry. If
you are seeing shaping, but something different than you
expected, it is probably because link characteristics are one
way, and you're measuring them over the round trip.
For instance, if you asked for a link that was 100Mbps, 30ms,
with 5% (0.05) packet loss rate (plr), you may expect ping to
show 30ms ping times and 5% loss rate. But what you should see
is 60ms latency for the round trip, and a loss rate of
9.75%. Latencies can be added, therefore 30ms + 30ms gives
60ms. However, loss rates are probabilities, and must be
multiplied. The chance of a packet making it across a 5% lossy
link is 95%, so with a 95% chance of arriving at the
destination, and a 95% chance of returning if it made it there,
and the total chance of making a round trip is .95 * .95 = .9025 or
90.25%, or a round trip loss rate of 9.75% on a 5% lossy link.
-
I decreased the bandwidth on a link and now the
ping time between the nodes has increased. Why?
Short answer: Decreasing the bandwidth of a link means that your
bytes take longer to get where they are going!
Long Answer: A ping packet is 98 bytes of data; 56 bytes of data
plus 8 bytes of ICMP header plus 20 bytes of IP header plus 14
bytes of ethernet header. At 100Mbs those 98 bytes takes .0078ms
to traverse the wire, which is hardly noticeable! If you have set
the delay of your link to 10ms, then your ping packets will incur
10ms+0.0078ms of delay in each direction, for a 20ms roundtrip time.
Say you set the bandwith of your link to 250Kbs. The wire time for
those same 98 bytes is now 3ms. If your delay is 10ms like above,
then your ping packets will incur 10ms+3ms of delay in each
direction, for a 26ms roundtrip time! If you set the bandwith to
100Kbs, the wire time is now 7.8ms and your ping packets will
incur 10ms+7.8ms of delay in each direction, for a 35.6ms roundtrip
time!
Note: If you have a router connecting two nodes, then each of
the two links will incur the same wire time (and delay of
course). In the above 250Kbs example, each ping packet will incur
3ms of wire time to the router and another 3ms from the
router to the destination. The ping reply packet will see the same
6ms of wire time. If your delay is again 10ms, then the ping
roundtrip is 52ms.
-
I wrote a small TCP application to test the
bandwidth of a link/LAN. I do not observe the bandwidth that I
asked for. Why?
Short answer: TCP needs large send and receive socket buffers in
order for its throughput to approach the capacity of long fat
networks (LFN) i.e. link/LANs with a large bandwidth-delay product
(BDP). Use UDP instead, if your intention is just to test the
bandwidth. If you need to tune the throughput of your TCP
application, refer to "Enabling High
Performance Data Transfers".
Long Answer: In order to observe the bandwidth that you specify,
it is necessary to keep the data pipe between the sender and the
receiver full. For a reliable window based protocol such as TCP,
the window size represents the number of unacknowledged bytes. TCP
needs to keep the unacknowledged bytes around until the acks for
them are received. These bytes are retained in socket buffers. On
a link/LAN with bandwidth B and round-trip-time (RTT) D, the
sender TCP needs to be able to transmit B times D bytes before
expecting any acknowledgement, if the data pipe has to be kept
full. The effective sending window is dependent on receiver
advertised window besides other things. It is necessary to have
sender and receiver socket buffers at least as high as BxD. If you
only care about optimal end-to-end TCP throughput, then the socket
buffers need to be BxD where B is the bandwidth of the bottleneck
link and D is the end-to-end RTT. Refer to the following "TCP
mini-tutorial" or a basic TCP/IP book for the gory details.
-
I wrote a small UDP application to test the
bandwidth of a link/LAN. I do not observe the bandwidth that I
asked for. Why?
Since UDP is unreliable and not flow controlled, you may just be
trying to push packets out on the wire too fast. In this case
packets will be dropped before even getting to the wire. You
may also be dropping packets at the receiver side if the consumer
cannot keep up.
If you are trying to saturate a 100Mb link and can't do it, it
may be due to using too small a packet size.
For small packet sizes, the limitation on a 100Mb link will be
the packet rate, not the raw bandwidth. The eepro100
ethernet NICs in most of our machines can only generate a little
over 100,000 packets per second (pps). With 64-byte packets,
you will thus only see about 51.2Mb/sec of raw data or about
14.4Mb/sec of UDP payload (a 64 byte ethernet packet can hold
only 18 bytes of UDP payload). Note that the theoretical max
is less than 150,000 pps for 64-byte packets, so even better NICs
would not change this.
Considering that stock FreeBSD and Linux can generate even fewer
packets per second due to interrupt and scheduling overheads, you
probably need to use at least 200-byte packets to saturate a 100Mb
link.
-
I am running a routing daemon in my topology
but the daemon claims that all my nodes are directly connected!
Why?
The routing daemons are probably talking to each other via the control
net and routing traffic through it since it is the shortest path.
You will need to configure your daemon to ignore the control net interface.
See the
control net section
of the tutorial.
|
