Neptune Jobs and Experiments

Table of Contents

Overview

In this section, you will learn about Neptune jobs and experiments and concepts related to them. You will also learn how to utilize Neptune’s features in your code, using a Client Library.

Jobs

Introduction

A job is a program registered for execution within Neptune. Every job consists of source code and configuration files defining metadata of the job.

Job’s metadata is divided in three groups: metadata defined by the user before creation of the job, metadata defined by the user in a job’s code and additional metadata provided by Neptune after job creation and during its execution.

Metadata defined by the user before creation of the job includes:

Metadata defined by the user in job’s code includes:

Metadata provided by Neptune after the creation of a job consists of:

Examples

Obtaining a Job Object
# Create a Context object representing a connection to Neptune.
ctx = neptune.Context()
# A Job object is accessible from Context.
job = ctx.job

The R programming interface for Neptune doesn’t feature a job object.

import io.deepsense.neptune.clientlibrary.NeptuneContextFactory;
import io.deepsense.neptune.clientlibrary.models.Job;
import io.deepsense.neptune.clientlibrary.models.NeptuneContext;

public static void main(String[] args) {
    NeptuneContext context = NeptuneContextFactory.createContext(args);
    Job job = context.getJob();
}
Accessing Job Information
# Print the job's ID.
print(ctx.job.id)
# Print the job's storage path.
print(ctx.storage_url)
# Print the job's ID.
print(jobId())
# Print the job's storage path.
print(storageUrl())
// Print the job's ID.
System.out.println(context.getJob().getId());
// Print the job's storage path.
System.out.println(context.getStorageUrl());

States

Introduction

Each job can be in one of the states listed below:

Once a job is started, it is no longer “queued” and becomes “running”. If the program finishes successfully it becomes “succeeded”, otherwise it becomes “failed”. Execution of queued and running jobs can be aborted, and then they change state to “aborted”.

In addition, each running job can occasionally become “not responding”. When a job is “not responding”, it means that there are communication problems between the job and Neptune. The communication problems can be temporary, meaning that after some time the job can become “responding” again.

Parameters

Introduction

Parameters are a set of user-defined variables that will be passed to the job’s program. Job’s parameters are defined in the configuration file. Parameters’ values can be passed to a job using command line parameters when enqueuing or executing the job. Each parameter has:

Reference

Parameters are accessible from the Context.params field. Parameter values can be retrieved using two different notations - the object notation and the dictionary notation.

Examples

Accessing Values of Parameters
# Access with object notation.
print(ctx.params.x)
# Access with dict-like notation.
print(ctx.params['x'])
# Access parameters.
print(params("x"))
// Access boolean parameters.
boolean b = context.getParams().get("b").getValue().asBoolean().get();

// Access double parameters.
double d = context.getParams().get("d").getValue().asDouble().get();

// Access integer parameters.
int i = context.getParams().get("i").getValue().asInteger().get();

// Access string parameters.
String s = context.getParams().get("s").getValue().asString().get();

// Handle optional parameters.
Optional<Integer> optional = context.getParams().get("optional").getValue().asInteger();

Channels

Introduction

A channel is a named series of two-dimensional points belonging to a job. Each point’s abscissa (point’s X) is always a floating point number. On the other hand, the ordinate (point’s Y) type depends on channel’s type, which can be either:

The points, also called channel’s values, represent a function, so X-coordinates have to be unique in a channel. Moreover, the points generated by jobs during execution must be in order so that the X-coordinates increase.

Reference

Job.create_channel()

Method create_channel() creates a new channel with given name, type and optional extra parameters:

ctx.job.create_channel(
    name,
    channel_type,
    is_history_persisted=True)
  • name: str/unicode

A channel name. It must be unique in the scope of a specific job.

  • channel_type: ChannelType

Type of the channel, one of:

  • neptune.ChannelType.NUMERIC

    Numeric channel, accepts float values.

  • neptune.ChannelType.TEXT

    Text channels, accepts str values.

  • neptune.ChannelType.IMAGE

    Image channel, accepts neptune.Image values.

  • is_history_persisted: bool

If True, all values sent to the channel are saved on Neptune Server, otherwise only the last value is saved.

Channel.send()

Method send() sends a channel value to Neptune Server.

channel.send(x, y)
  • x: float

The value of channel value’s X-coordinate. Multiple calls of send() in a job have to be made in an order of increasing values of x.

  • y (type depending on the channel’s type)

The value of channel value’s Y-coordinate.

Channel’s type y’s type
NUMERIC float
TEXT str/unicode
IMAGE neptune.Image
neptune.Image

Objects of class neptune.Image represent information about images sent to image channels.

class Image(...):
    def __init__(self, name, description, data):
        ...
  • name: str/unicode

Name of the image, displayed in the Channels tab on job’s dashboard.

  • description: str/unicode

Description of the image, displayed in the Channels tab on job’s dashboard.

Image data in PIL.Image format.

NOTE: The maximum supported size of the image is 512x512 px.

createNumericChannel(), createTextChannel()

Functions createNumericChannel() and createTextChannel() create channels with a given name and optional extra parameters:

# Creates a numeric channel.
createNumericChannel(channelName, isHistoryPersisted = TRUE, isLastValueExposed = TRUE)

# Create a text channel.
createTextChannel(channelName, isHistoryPersisted = TRUE, isLastValueExposed = TRUE)
  • channelName: character

A channel name. It must be unique in the scope of a specific job.

  • isHistoryPersisted: logical

If TRUE, all values sent to the channel are saved on Neptune Server, otherwise only the last value is saved.

  • isLastValueExposed: logical

If TRUE, the last channel value can be displayed on the job list.

createImageChannel()

Function createImageChannel() creates a new image channel with given name:

createNumericChannel(channelName)
  • channelName: character

A channel name. It must be unique in the scope of a specific job.

channelSend()

Function channelSend() sends a channel value to Neptune Server.

channelSend(channelName, x, y)
  • channelName: character

The channel name.

  • x: numeric

The value of channel value’s X-coordinate. Multiple calls of channelSend() in a job have to be made in an order of increasing values of x.

  • y (type depending on the channel’s type)

The value of channel value’s Y-coordinate.

Channel’s type y’s type
numeric numeric
text character
image neptuneImage

neptuneImage()

Function neptuneImage() creates a representation of an image that can be sent to a channel.

neptuneImage(name, description, data)
  • name: character

Name of the image, displayed in the Channels tab on job’s dashboard.

  • description: character

Description of the image, displayed in the Channels tab on job’s dashboard.

  • data: array

Image data as a numeric array of the dimensions: height x width x channels. The standard RGB format (3 channels) is supported. The numeric values in the array are reals between 0 and 1.

NOTE: The maximum supported size of the image is 512x512 px.

Job.createNumericChannel(), Job.createTextChannel()

Methods Job.createNumericChannel() and Job.createTextChannel() create channels with a given name and optional extra parameters:

// Create a numeric channel.
Channel<Double> createNumericChannel(String name);
Channel<Double> createNumericChannel(String name, boolean isHistoryPersisted, boolean isLastValueExposed);

// Create a text channel.
Channel<String> createTextChannel(String name);
Channel<String> createTextChannel(String name, boolean isHistoryPersisted, boolean isLastValueExposed);
  • name: String

A channel name. It must be unique in the scope of a specific job.

  • isHistoryPersisted: boolean

If true, all values sent to the channel are saved on Neptune Server, otherwise only the last value is saved.

  • isLastValueExposed: boolean

If true, the last channel value can be displayed on the job list.

Job.createImageChannel()

Method Job.createImageChannel() creates a new image channel with given name:

Channel<NeptuneImage> createImageChannel(String name);
  • name: String

A channel name. It must be unique in the scope of a specific job.

Channel.send()

Method Channel.send() sends a channel value to Neptune Server.

import io.deepsense.neptune.clientlibrary.models.Channel;
...

Channel<Double> numericChannel = job.createNumericChannel("numericChannel");
numericChannel.send(1.0, 2.5);
  • x: double

The value of channel value’s X-coordinate. Multiple calls of Channel.send() in a job have to be made in an order of increasing values of x.

  • y (type depending on the channel’s type)

The value of channel value’s Y-coordinate.

Channel’s type y’s type
numeric Double
text String
image NeptuneImage

class NeptuneImage

Class NeptuneImage is a representation of an image that can be sent to a channel.

package io.deepsense.neptune.clientlibrary.models;

public class NeptuneImage {
    public NeptuneImage(String name, String description, BufferedImage data) { ... }
    ...
}
  • name: String

Name of the image, displayed in the Channels tab on job’s dashboard.

  • description: String

Description of the image, displayed in the Channels tab on job’s dashboard.

Image data as an object of the java.awt.BufferedImage class.

NOTE: The maximum supported size of the image is 512x512 px.

Examples

Creating Numeric and Text Channels
numeric_channel = ctx.job.create_channel(
    name='numeric_channel',
    channel_type=neptune.ChannelType.NUMERIC)

text_channel = ctx.job.create_channel(
    name='text_channel',
    channel_type=neptune.ChannelType.TEXT)

numeric_channel.send(x=1, y=2.5)
numeric_channel.send(x=1.5, y=5)

text_channel.send(x=2.5, y='text 1')
text_channel.send(x=3, y='text 2')
createNumericChannel("numericChannel")
createTextChannel("textChannel")

channelSend("numericChannel", x = 1, y = 2.5)
channelSend("numericChannel", x = 1.5, y = 5)

channelSend("textChannel", x = 2.5, y = "text 1")
channelSend("textChannel", x = 3, y = "text 2")
import io.deepsense.neptune.clientlibrary.models.Channel;
...

Channel<Double> numericChannel = job.createNumericChannel("numericChannel");
Channel<String> textChannel = job.createTextChannel("textChannel");

numericChannel.send(1, 2.5);
numericChannel.send(1.5, 5);

textChannel.send(2.5, "text 1");
textChannel.send(3, "text 2");
Creating an Image Channel
channel = job.create_channel(
    name='image_channel',
    channel_type=neptune.ChannelType.IMAGE)

channel.send(
    x=1,
    y=neptune.Image(
        name='#1 image name',
        description='#1 image description',
        data=Image.open("/home/ubuntu/image1.jpg")))
library(jpeg)
...

createImageChannel("imageChannel")

channelSend("imageChannel",
    x = 1,
    y = neptuneImage(
        "#1 image name",
        "#1 image description",
        readJPEG("/home/ubuntu/image.jpg")))
import io.deepsense.neptune.clientlibrary.models.Channel;
import io.deepsense.neptune.clientlibrary.models.NeptuneImage;
...

Channel<NeptuneImage> imageChannel = context.getJob().createImageChannel("imageChannel");

NeptuneImage image = new NeptuneImage(
    "#1 image name",
    "#1 image description,
    ImageIO.read("/home/ubuntu/image.jpg"));

neptuneImage.send(1, image);

Charts

Introduction

A chart is a named group of series. Each series consists of a channel and a channel label which will be used to display channel values. Series of one chart are displayed together on the same diagram.

Reference

Job.create_chart()

Method create_chart() creates a new chart that groups values of one or more numeric channels:

ctx.job.create_chart(name, series)
  • name: str/unicode

Unique chart name.

  • series: dict or list[Series]

Either a dictionary representing series of the chart or a list with the neptune.Series objects. If the param is a dictionary, the keys are names of the series and the values are objects representing channels returned by Job.create_channel().

Chart.add_series()

Method add_series() adds a new series to the chart:

my_chart.add_series(name, channel, series_type)
  • name: str/unicode

The name of the series.

  • channel: Channel

The channel providing values for the new series.

  • series_type: SeriesType

Either SeriesType.LINE (the consecutive points on the series are connected with lines) or SeriesType.DOT (the points are not connected). The default value is SeriesType.LINE.

createChart()

Function createChart() creates a new chart that groups values of one or more numeric channels:

createChart(chartName, series)
  • name: character

Unique chart name.

  • series: list

A list with definitions of series. You can define series with:

  • a list containing the channel name and the series type,
  • a channel name (the series type will be LINE).

Series type can be either “LINE” or “DOT”.

addSeries()

Function addSeries() adds a new series to a chart:

addSeries(chart, name, channel, type)
  • chart: character

The name of the chart the series is added to.

  • name: character

The name of the series.

  • channel: character

The name of the channel providing values for the new series.

  • type: character

Either “LINE” (the consecutive points on the series are connected with lines) or “DOT” (the points are not connected). The default value is “LINE”.

Job.createChart()

Method Job.createChart() creates a new chart that groups values of one or more numeric channels:

Chart createChart(String name, Iterable<ChartSeries> seriesCollection);
  • name: String

Unique chart name.

  • seriesCollection: Iterable<ChartSeries>

An iterable of chart’s series. Each series corresponds to a channel.

Chart.addSeries()

Method Chart.addSeries() adds a new series to the chart:

void addSeries(String name, Channel channel);
void addSeries(String name, Channel channel, ChartSeriesType type);
  • name: String

The name of the series.

  • channel: Channel

The channel providing values for the new series.

  • type: ChartSeriesType

Either ChartSeriesType.LINE (the consecutive points on the series are connected with lines) or ChartSeriesType.DOT (the points are not connected). The default value is ChartSeriesType.LINE.

Examples

Creating a Chart
ch1 = ctx.job.create_channel(
    name='ch1',
    channel_type=neptune.ChannelType.NUMERIC)
ch2 = ctx.job.create_channel(
    name='ch2',
    channel_type=neptune.ChannelType.NUMERIC)

ctx.job.create_chart(
    name='comparison of ch1 & ch2',
    series={'channel 1': ch1, 'channel 2': ch2})

ctx.job.create_chart(
    name='another comparison of ch1 & ch2',
    series=[
        neptune.Series('channel 1', ch1, neptune.SeriesType.LINE),
        neptune.Series('channel 2', ch2, neptune.SeriesType.DOT)
    ])

yet_another_chart = ctx.job.create_chart(
    name='yet another comparison of ch1 & ch2',
    series={})

yet_another_chart.add_series('channel 1', ch1)
yet_another_chart.add_series('channel 2', ch2, neptune.SeriesType.DOT)
createNumericChannel("ch1")
createNumericChannel("ch2")

createChart(
    chartName = "comparison of ch1 & ch2",
    series = list("channel 1" = "ch1", "channel 2" = "ch2"))

createChart(
    chartName = "another comparison of ch1 & ch2",
    series = list(
        "channel 1" = list(channel = "ch1", type = "LINE"),
        "channel 2" = list(channel = "ch2", type = "DOT")))

createChart(
    chartName = "yet another comparison of ch1 & ch2",
    series = list())

addSeries(
    chart = "yet another comparison of ch1 & ch2",
    name = "channel 1",
    channel = "ch1",
    type = "LINE")

addSeries(
    chart = "yet another comparison of ch1 & ch2",
    name = "channel 2",
    channel = "ch2",
    type = "DOT")
import io.deepsense.neptune.clientlibrary.models.Channel;
import io.deepsense.neptune.clientlibrary.models.Chart;
import io.deepsense.neptune.clientlibrary.models.ChartSeries;
import io.deepsense.neptune.clientlibrary.models.ChartSeriesType;
...

Channel<Double> ch1 = job.createNumericChannel("ch1");
Channel<Double> ch2 = job.createNumericChannel("ch2");

Collection<ChartSeries> seriesCollection = new ArrayList<>();
seriesCollection.add(new ChartSeries("channel 1", ch1));
seriesCollection.add(new ChartSeries("channel 2", ch2));

job.createChart("comparison of ch1 & ch2", seriesCollection);

Collection<ChartSeries> anotherSeriesCollection = new ArrayList<>();
anotherSeriesCollection.add(
    new ChartSeries("channel 1", ch1, ChartSeriesType.LINE));
anotherSeriesCollection.add(
    new ChartSeries("channel 2", ch2, ChartSeriesType.DOT));

job.createChart("another comparison of ch1 & ch2", anotherSeriesCollection);

Collection<ChartSeries> yetAnotherSeriesCollection = new ArrayList<>();

Chart yetAnotherChart = job.createChart(
                            "yet another comparison of ch1 & ch2",
                            yetAnotherSeriesCollection);

yetAnotherChart.addSeries("channel 1", ch1);
yetAnotherChart.addSeries("channel 2", ch2, ChartSeriesType.DOT);

Actions

Introduction

An Action is a registered function that can be invoked externally with an argument. Action can be invoked either via the CLI or using the Web UI. Both argument and action result cannot exceed 160 characters.

NOTE: Actions are executed in a separate thread, therefore your code should be thread-safe.

Reference

Job.register_action()

Method register_action() registers a new action that calls a function with the argument provided on invocation:

ctx.job.register_action(name, handler)
  • name: str/unicode

Unique action name.

  • handler: function(arg: str) -> str

A single-argument function that will be called on an action invocation. The handler must take one unicode/str argument and return unicode/str as the result.

Examples

Creating an Action
session = ...

def save_model(path):
    return str(session.save_model(path))

ctx.job.register_action(name='save model', handler=save_model)
ctx.job.register_action(name='get session param', handler=lambda param_name: session[param_name])

Actions are not yet supported in the R Client Library.

Actions are not yet supported in the Java Client Library.

Tags

Introduction

A tag is a word that contains only lowercase letters, numbers, underscores and dashes. Tags can be assigned to a job or removed from a job anytime throughout job’s lifetime. When listing jobs, it is possible to search by tags, so tags are useful to mark jobs. Job’s tags can be set from both the configuration file and the job’s code or by command line parameters.

Reference

Tags are accessible from the Job.tags field which exposes a MutableSequence interface.

Tags are accessible with 3 functions:

  • tags() - returns the list of all tags of the job,
  • addTags(...) - adds new tags to the job,
  • removeTags(...) - removes tags from the job.

Tags are accessible through the Job.getTags() method. It returns an instance of Set<String> that allows to browse, add, and remove tags.

Examples

Accessing Tags
my_tag_exists = 'my-tag' in ctx.job.tags
myTagExists <- is.element("my-tag", tags())
boolean myTagExists = job.getTags().contains("my-tag");
Modifying Tags
ctx.job.tags.append('new-tag')
ctx.job.tags.remove('new-tag')
ctx.job.tags += ['tag1', 'tag2', 'tag3']
addTags("new-tag", "another-new-tag")
removeTags("old-tag", "another-new-tag")
Tags tags = job.getTags();
job.getTags().addAll(Arrays.asList("new-tag", "another-new-tag"));
job.getTags().remove("old-tag");

Properties

Introduction

A property is a key-value pair of two strings used to store additional metadata with a job. They can be used to store custom data about the job, e.g. the hardware parameters of the machine where the job was executed: number of CPU and GPU cores, amount of RAM, etc. Job’s properties can be set from the configuration file and job’s code or by command line parameters.

Reference

Properties are accessible from the Job.properties field which exposes a MutableMapping interface.

In the R client library, properties are represented as lists of values named with their keys.

Properties are accessible with 3 functions:

  • properties() - returns the list of all properties of the job,
  • addProperties(...) - adds new properties to the job,
  • removeProperties(...) - removes properties from the job.

Properties are accessible through the Job.getProperties() method. It returns an instance of Map<String, String> that allows to browse, add, and remove properties.

Examples

Accessing Properties
print(ctx.job.properties['my-property'])
print(properties()[["my-property"]])
System.out.println(job.getProperties().get("my-property"));
Modifying Properties
ctx.job.properties['property1'] = 'new-value'
ctx.job.properties['property2'] = 'new-value'
del ctx.job.properties['property2']
addProperties(property1 = "new-value", property2 = "new-value")
removeProperties("property2")
Properties properties = job.getProperties();
properties.put("property1", "new-value");
properties.put("property2", "new-value");
properties.remove("property2");

Requirements

Requirements are a list of labels that can be used to define demands regarding execution environment. You can specify requirements in the configuration file and in the enqueue command. Requirements are useful when dealing with remote job execution.

Experiments

Introduction

An experiment is an abstraction that represents hyperparameter optimization. Currently Neptune supports only grid search method. You can create an experiment by passing a range or a list of values instead of specific values for numeric parameters. Neptune will run a grid search for specified parameters. For every combination of parameter values Neptune will run a job and group them in the experiment.

States

Introduction

Each experiment can be in one of the states listed below:

Metric

Introduction

A metric is a measure against which we want to optimize algorithm’s performance. Every job in the experiment has to report value for the metric. The metric is used to compare jobs in the experiment and select the best job. You can define a metric in a configuration file. For every metric you have to provide:

Using the metric Neptune will determine the best job in the experiment.

See Diabetes Progression Prediction example for further details on how to perform a grid search experiment.

Reference

Metric info is represented as an object of the Metric class. It is accessible via Context.metric. When a metric is not specified, the Context.metric field is None.

Definition of the Metric class:

class Metric(object):
    def __init__(self, channel_name, direction):
        self.channel_name = channel_name
        self.direction = direction
  • channel_name: str/unicode - name of a numeric channel with metric values,
  • direction: str - either “minimize” or “maximize”.

Metric info is accessible via metricChannelName() and metricDirection() functions. When a metric is not specified, the functions return NULL.

Description of the metric-related functions:

  • metricChannelName(): character - name of a numeric channel with metric values,
  • metricDirection(): character - either “minimize” or “maximize”.

Metric info is accessible via the NeptuneContext.getMetric() method, returning Optional<Metric> type. When a metric is not specified, the method returns an empty Optional.

Definition of the Metric type:

public interface Metric {

    String getChannelName();

    MetricDirection getMetricDirection();
}
  • String getChannelName() - name of a numeric channel with metric values,
  • MetricDirection getMetricDirection() - either “minimize” or “maximize”.

Examples

Creating a channel for metric values
metric_channel = ctx.job.create_channel(
    name=ctx.metric.channel_name,
    channel_type=neptune.ChannelType.NUMERIC)
createNumericChannel(metricChannelName())
String metricChannelName = context.getMetric().get().getChannelName();
job.createNumericChannel(metricChannelName);
Printing metric info
print(ctx.metric.channel_name)
print(ctx.metric.direction)
print(metricChannelName())
print(metricDirection())
Metric metric = context.getMetric().get();
System.out.println(metric.getChannelName());
System.out.println(metric.getDirection());