Getting Started

Table of Contents

This tutorial gives a quick introduction to Neptune’s features. We will show how to create a simple experiment and how to run it as a Neptune job. We will go through the job’s code and the configuration file. Also, we will describe how the job’s parameters work, and explain how to pass parameters to a job. Finally, we will enqueue and run a couple of jobs with different parameters, and show how to monitor the jobs’ statuses and analyze execution history using the Web UI.

Get Access to Neptune

To run the following examples, you need access to Neptune Server. You can create an account at Neptune Go, which will enable you to run your own experiments, as well as explore other people’s work. All experiments performed using Neptune Go are public which lets you have a taste of cooperative style that Neptune is designed for. Due to the public nature of Neptune Go, the number of experiments per user is limited.

Alternatively you can join our Early Adopters Program to get your personal instance of Neptune, available as a service.

If you prefer an on-premise Neptune instance, contact us.

Install the Required Libraries

Before you can use Neptune, you’ll need to install Neptune CLI and the Python Client Library. To download and configure Neptune CLI and the Client Library, follow instructions from the Downloads section.

Before you can use Neptune, you’ll need to install Neptune CLI and the R Client Library. To download and configure Neptune CLI and the Client Library, follow instructions from the Downloads section.

Before you can use Neptune, you’ll need to install Neptune CLI and the Java Client Library. To download and configure Neptune CLI and the Client Library, follow instructions from the Downloads section.

To run the Java code from this tutorial, you also need to have Gradle 2.9+ installed.

Create a New Job

Our goal is to create a simple parameterizable job that provided with amplitude and sampling_rate generates sine and cosine as functions of time (in seconds).

In order to have a good starting point, we prepared sample code and a configuration file on GitHub.

You can clone the neptune-examples repository with Git:

git clone -b 1.5 https://github.com/deepsense-io/neptune-examples.git
cd neptune-examples/python/getting-started

or just download a ZIP archive containing the examples.

wget https://github.com/deepsense-io/neptune-examples/archive/1.5.zip
unzip 1.5.zip
cd neptune-examples-1.5/python/getting-started

In order to have a good starting point, we prepared sample code and a configuration file on GitHub.

You can clone the neptune-examples repository with Git:

git clone -b 1.5 https://github.com/deepsense-io/neptune-examples.git
cd neptune-examples/r/getting-started

or just download a ZIP archive containing the examples.

wget https://github.com/deepsense-io/neptune-examples/archive/1.5.zip
unzip 1.5.zip
cd neptune-examples-1.5/r/getting-started

In order to have a good starting point, we prepared sample code and a configuration file on GitHub.

You can clone the neptune-examples repository with Git:

git clone -b 1.5 https://github.com/deepsense-io/neptune-examples.git
cd neptune-examples/java/getting-started

or just download a ZIP archive containing the examples.

wget https://github.com/deepsense-io/neptune-examples/archive/1.5.zip
unzip 1.5.zip
cd neptune-examples-1.5/java/getting-started

Job’s Configuration File

The file named neptune.yaml is our job’s configuration file. Besides providing basic information about the job, it may define a sequence of parameters that will make our sine-cosine generator customizable.

name: Sine-Cosine Generator
project: Trigonometry
parameters:
  - name: amplitude
    type: double
    default: 1.0
    required: false
  - name: sampling_rate
    type: double
    default: 2
    required: false

Here we have defined the amplitude and sampling_rate parameters of type double with default values 1.0 and 2 respectively. These parameters are now accessible within our code. Additionally their values can be overwritten for every execution of our code.

Let’s Get Through the Code!

The main.py file contains the source code of a simple parameterizable code that interacts with Neptune.

The first lines in the source file contain imports of the required libraries, including Neptune Python Client Library:

import math
import time

from deepsense import neptune

In the next step, a Context object is created as an endpoint for communication with Neptune:

ctx = neptune.Context()

The job has two parameters: amplitude and sampling_rate, declared previously in neptune.yaml. Parameters’ values are accessible in the code through the Context object just created:

amplitude = ctx.params.amplitude
sampling_rate = ctx.params.sampling_rate

The next part of the code contains declarations of channels. They enable communication of our job with Neptune. Sine and cosine values will be sent to separate NUMERIC channels:

sin_channel = ctx.job.create_channel(name='sin', channel_type=neptune.ChannelType.NUMERIC)
cos_channel = ctx.job.create_channel(name='cos', channel_type=neptune.ChannelType.NUMERIC)

A TEXT channel is created to log formatted points of sine and cosine calculated in each iteration:

logging_channel = ctx.job.create_channel(name='logging', channel_type=neptune.ChannelType.TEXT)

Values from sin_channel and cos_channel are plotted on a single chart that will be visible on the job’s dashboard. The method ctx.job.create_chart() is responsible for creating the chart:

ctx.job.create_chart(name='sin & cos chart', series={'sin': sin_channel, 'cos': cos_channel})

The remaining part of the code computes the values of sine and cosine functions and sends them through channels to Neptune. Channel values are sent to Neptune with the Channel.send() method.

# The time interval between samples.
period = 1.0 / sampling_rate
# The initial timestamp, corresponding to x = 0 in the coordinate axis.
zero_x = time.time()

iteration = 0

while True:
    iteration += 1

    # Computes the values of sine and cosine.
    now = time.time()
    x = now - zero_x
    sin_y = amplitude * math.sin(x)
    cos_y = amplitude * math.cos(x)

    # Sends the computed values to the defined numeric channels.
    sin_channel.send(x=x, y=sin_y)
    cos_channel.send(x=x, y=cos_y)

    # Formats a logging entry.
    logging_entry = "sin({x})={sin_y}; cos({x})={cos_y}".format(x=x, sin_y=sin_y, cos_y=cos_y)

    # Sends a logging entry.
    logging_channel.send(x=iteration, y=logging_entry)

    time.sleep(period)

The main.R file contains the source code of a simple parameterizable code that interacts with Neptune.

The first line in the source file contains an import of Neptune R Client Library:

library(neptune)

The job has two parameters: amplitude and sampling_rate, declared previously in neptune.yaml. Parameters’ values are accessible in the code:

samplingRate <- params("sampling_rate")
amplitude <- params("amplitude")

The next part of the code contains declarations of channels. They enable communication of our job with Neptune. Sine and cosine values will be sent to separate numeric channels:

createNumericChannel("sin")
createNumericChannel("cos")

A text channel is created to log formatted points of sine and cosine calculated in each iteration:

createTextChannel("logging")

Values from sin and cos channels are plotted on a single chart that will be visible on the job’s dashboard. The method createChart() is responsible for creating the chart:

createChart(chartName = "sin & cos chart", series = list("my_sin" = "sin", "my_cos" = "cos"))

The remaining part of the code computes the values of sine and cosine functions and sends them through channels to Neptune. Channel values are sent to Neptune with the channelSend() method.

period <- 1.0 / samplingRate
zeroX <- unclass(Sys.time())

iteration <- 0

while (TRUE) {
  iteration <- iteration + 1

  now <- unclass(Sys.time())
  x <- now - zeroX

  sinY <- amplitude * sin(x)
  cosY <- amplitude * cos(x)

  channelSend("sin", x, sinY)
  channelSend("cos", x, cosY)

  loggingEntry <- paste("sin(", x, ")=", sinY, "; cos(", x, ")=", cosY)
  channelSend("logging", iteration, loggingEntry)

  Sys.sleep(period)
}

The GettingStarted.java file contains the source code of a simple parametrizable code that interacts with Neptune.

The first lines in the source file contain imports of the required libraries, including Neptune Java Client Library:

package io.deepsense.neptune.gettingstarted;

import io.deepsense.neptune.clientlibrary.NeptuneContextFactory;
import io.deepsense.neptune.clientlibrary.models.Channel;
import io.deepsense.neptune.clientlibrary.models.ChartSeries;
import io.deepsense.neptune.clientlibrary.models.NeptuneContext;

import java.util.ArrayList;
import java.util.Collection;

In the first line of the main method, a NeptuneContext object is created as an endpoint for communication with Neptune.

public class GettingStarted {
    public static void main(String[] args) throws InterruptedException {
        NeptuneContext context = NeptuneContextFactory.createContext(args);
        ...
    }
}

The job has two parameters: amplitude and sampling_rate, declared previously in neptune.yaml. Parameters’ values are accessible in the code:

double amplitude = context.getParams().get("amplitude").getValue().asDouble().get();
double samplingRate = context.getParams().get("sampling_rate").getValue().asDouble().get();

The next part of the code contains declarations of channels. They enable communication of our job with Neptune. Sine and cosine values will be sent to separate numeric channels:

Channel<Double> sinChannel = context.getJob().createNumericChannel("sin");
Channel<Double> cosChannel = context.getJob().createNumericChannel("cos");

A text channel is created to log formatted points of sine and cosine calculated in each iteration:

Channel<String> loggingChannel = context.getJob().createTextChannel("logging");

Values from sinChannel and cosChannel are plotted on a single chart that will be visible on the job’s dashboard. The chart’s series are defined in the next few lines:

Collection<ChartSeries> seriesCollection = new ArrayList<>();
seriesCollection.add(new ChartSeries("sin", sinChannel));
seriesCollection.add(new ChartSeries("cos", cosChannel));

The method createChart() is responsible for creating the chart:

context.getJob().createChart("sin & cos chart", seriesCollection);

The remaining part of the code computes the values of sine and cosine functions and sends them through channels to Neptune. Channel values are sent to Neptune with the send() method.

long period = (long) (1000.0 / samplingRate);

double zeroX = System.nanoTime() / 1e9;

int iteration = 0;

while (true) {
    iteration += 1;

    double x = System.nanoTime() / 1e9 - zeroX;

    double sinY = amplitude * Math.sin(x);
    double cosY = amplitude * Math.cos(x);

    sinChannel.send(x, sinY);
    cosChannel.send(x, cosY);

    loggingChannel.send(
        iteration,
        String.format("sin(%f)=%f; cos(%f)=%f",
            x, sinY, x, cosY));

    Thread.sleep(period);
    }

}

Running the Job

To run the job on your local machine, use the neptune run command:

neptune run

This command executes main.py with configuration from neptune.yaml.

To run the job on your local machine, use the neptune run command:

neptune run

This command executes main.R with configuration from neptune.yaml.

To run the job on your local machine, you need to compile the Java code and put the compiled code in a JAR file:

$ gradle wrapper
$ ./gradlew uberjar

Next, copy the JAR file to your current working directory:

$ cp build/libs/getting-started.jar .

and execute the neptune run command.

neptune run getting-started.jar

This command executes getting-started.jar with configuration from neptune.yaml.

Execution of neptune run should result in output similar to this:

>
> Job enqueued, id: 2a1033cc-8ff9-4a66-a381-1d5f5dea68cd
>
> To browse the job, follow:
> http://[your Neptune IP address]/#dashboard/job/2a1033cc-8ff9-4a66-a381-1d5f5dea68cd
>

Viewing the Job’s Dashboard

Follow the link from the first lines of output from the neptune run command to view the job’s dashboard with all its details:

Main Dashboard of the ‘Sine-Cosine Generator’ Job

As you can see, our chart updates in real time for every calculated point.

If you take a look at the code we had run, you will see that the chart named sin & cos chart is a composition of two independent numeric channels named sin and cos. To preview them, go to the Channels tab. You can see both numeric channels for sin and cos and a logging channel of text type. To view it’s last entries, click on it:

Channels of the ‘Sine-Cosine Generator’ Job

Aborting the Job

To abort our job, which generates channel values infinitely, click Abort on the job’s dashboard or send an interrupt signal with Ctrl-C to the terminal you had run the job from. Our spawned process will exit and our job’s state will be changed to aborted. At this point, no channel can be updated and all channel values already sent are stored for further analysis.

Side Effects of the Job

Neptune executes the job inside a storage (~/neptune-storage by default). When you run a job using Neptune CLI, it creates a directory dedicated for this particular job’s execution inside the storage.

All the content of the directory from which you run a job is copied to the storage. In our example, once the job is finished, the storage will contain a source file, a configuration file and files containing job’s output from stdout and stderr.

Playing With the Parameters

The “Sine-Cosine Generator” job has two parameters. Their values can be altered by specifying command line parameters while invoking Neptune commands: neptune run, neptune enqueue, neptune exec. Let’s enqueue the job a couple of times using different values of parameters.

Let’s set the amplitude to 5 and sampling_rate to 2.5 for the first job. We just need to add two command line parameters, separated from CLI’s parameters by a double dash.

neptune enqueue -- --amplitude 5 --sampling_rate 2.5
neptune enqueue -- --amplitude 5 --sampling_rate 2.5
neptune enqueue getting-started.jar -- --amplitude 5 --sampling_rate 2.5

For the second job, we set amplitude to 2 and sampling_rate to 0.5.

neptune enqueue -- --amplitude 2 --sampling_rate 0.5
neptune enqueue -- --amplitude 2 --sampling_rate 0.5
neptune enqueue getting-started.jar -- --amplitude 2 --sampling_rate 0.5

It’s not necessary to set all parameters, let’s only set the amplitude. The remaining parameter will be set to its default value.

neptune enqueue -- --amplitude 0.1
neptune enqueue -- --amplitude 0.1
neptune enqueue getting-started.jar -- --amplitude 0.1

Now, these three jobs should appear in the job list in the Web UI. You can go to the job’s parameter panel and read parameters’ values. You will notice that the parameters have new values - the one that were passed as the command line parameters.

Job’s parameters with overridden default values

Let’s execute the first enqueued job using the neptune exec command.

neptune exec [JOB ID]

You will see on the job list that one job is aborted (the one executed as the first one and later aborted). Two jobs will be queued and one will be running (the one we have just started).

Queued, running and aborted jobs on the job list

Read More

We hope you enjoyed your first experience with Neptune! We encourage you to read more about Neptune. Next to read: Examples.