Writing a reader

Authors:

Bradley Chambers, Scott Lewis

Contact:

brad.chambers@gmail.com

Date:

11/02/2017

PDAL’s command-line application can be extended through the development of reader functions. In this tutorial, we will give a brief example.

The header

First, we provide a full listing of the reader header.

 1// MyReader.hpp
 2
 3#pragma once
 4
 5#include <pdal/PointView.hpp>
 6#include <pdal/Reader.hpp>
 7#include <pdal/util/IStream.hpp>
 8
 9namespace pdal
10{
11  class MyReader : public Reader
12  {
13  public:
14    MyReader() : Reader() {};
15    std::string getName() const;
16
17  private:
18    std::unique_ptr<ILeStream> m_stream;
19    point_count_t m_index;
20    double m_scale_z;
21
22    virtual void addDimensions(PointLayoutPtr layout);
23    virtual void addArgs(ProgramArgs& args);
24    virtual void ready(PointTableRef table);
25    virtual point_count_t read(PointViewPtr view, point_count_t count);
26    virtual void done(PointTableRef table);
27  };
28}

1    std::unique_ptr<ILeStream> m_stream;
2    point_count_t m_index;
3    double m_scale_z;

m_stream is used to process the input, while m_index is used to track the index of the records. m_scale_z is specific to MyReader, and will be described later.

1    virtual void addDimensions(PointLayoutPtr layout);
2    virtual void addArgs(ProgramArgs& args);
3    virtual void ready(PointTableRef table);
4    virtual point_count_t read(PointViewPtr view, point_count_t count);
5    virtual void done(PointTableRef table);

Various other override methods for the stage. There are a few others that could be overridden, which will not be discussed in this tutorial.

Note

See ./include/pdal/Reader.hpp of the source tree for more methods that a reader can override or implement.

The source

Again, we start with a full listing of the reader source.

  1// MyReader.cpp
  2
  3#include "MyReader.hpp"
  4#include <pdal/util/ProgramArgs.hpp>
  5
  6namespace pdal
  7{
  8  static PluginInfo const s_info
  9  {
 10    "readers.myreader",
 11    "My Awesome Reader",
 12    "http://link/to/documentation"
 13  };
 14
 15  CREATE_SHARED_STAGE(MyReader, s_info)
 16
 17  std::string MyReader::getName() const { return s_info.name; }
 18
 19  void MyReader::addArgs(ProgramArgs& args)
 20  {
 21    args.add("z_scale", "Z Scaling", m_scale_z, 1.0);
 22  }
 23
 24  void MyReader::addDimensions(PointLayoutPtr layout)
 25  {
 26    layout->registerDim(Dimension::Id::X);
 27    layout->registerDim(Dimension::Id::Y);
 28    layout->registerDim(Dimension::Id::Z);
 29    layout->registerOrAssignDim("MyData", Dimension::Type::Unsigned64);
 30  }
 31
 32  void MyReader::ready(PointTableRef)
 33  {
 34    m_index = 0;
 35    SpatialReference ref("EPSG:4385");
 36    setSpatialReference(ref);
 37  }
 38
 39  template <typename T>
 40  T convert(const StringList& s, const std::string& name, size_t fieldno)
 41  {
 42      T output;
 43      bool bConverted = Utils::fromString(s[fieldno], output);
 44      if (!bConverted)
 45      {
 46          std::stringstream oss;
 47          oss << "Unable to convert " << name << ", " << s[fieldno] <<
 48              ", to double";
 49          throw pdal_error(oss.str());
 50      }
 51
 52      return output;
 53  }
 54
 55
 56  point_count_t MyReader::read(PointViewPtr view, point_count_t count)
 57  {
 58    PointLayoutPtr layout = view->layout();
 59    PointId nextId = view->size();
 60    PointId idx = m_index;
 61    point_count_t numRead = 0;
 62
 63    m_stream.reset(new ILeStream(m_filename));
 64
 65    size_t HEADERSIZE(1);
 66    size_t skip_lines((std::max)(HEADERSIZE, (size_t)m_index));
 67    size_t line_no(1);
 68    for (std::string line; std::getline(*m_stream->stream(), line); line_no++)
 69    {
 70      if (line_no <= skip_lines)
 71      {
 72        continue;
 73      }
 74
 75      // MyReader format:  X::Y::Z::Data
 76      StringList s = Utils::split2(line, ':');
 77
 78      unsigned long u64(0);
 79      if (s.size() != 4)
 80      {
 81        std::stringstream oss;
 82        oss << "Unable to split proper number of fields.  Expected 4, got "
 83            << s.size();
 84        throw pdal_error(oss.str());
 85      }
 86
 87      std::string name("X");
 88      view->setField(Dimension::Id::X, nextId, convert<double>(s, name, 0));
 89
 90      name = "Y";
 91      view->setField(Dimension::Id::Y, nextId, convert<double>(s, name, 1));
 92
 93      name = "Z";
 94      double z = convert<double>(s, name, 2) * m_scale_z;
 95      view->setField(Dimension::Id::Z, nextId, z);
 96
 97      name = "MyData";
 98      view->setField(layout->findProprietaryDim(name),
 99                     nextId,
100                     convert<unsigned int>(s, name, 3));
101
102      nextId++;
103      if (m_cb)
104        m_cb(*view, nextId);
105    }
106    m_index = nextId;
107    numRead = nextId;
108
109    return numRead;
110  }
111
112  void MyReader::done(PointTableRef)
113  {
114    m_stream.reset();
115  }
116
117} //namespace pdal

In your reader implementation, you will use a macro to create the plugin. This macro registers the plugin with the PDAL PluginManager. In this case, we are declaring this as a SHARED stage, meaning that it will be loaded at runtime instead of being linked to the main PDAL installation. The macro is supplied with the class name of the plugin and a PluginInfo object. The PluginInfo objection includes the name of the plugin, a description, and a link to documentation.

When making a shared plugin, the name of the shared library must correspond with the name of the reader provided here. The name of the generated shared object must be

libpdal_plugin_reader_<reader name>.<shared library extension>

1  static PluginInfo const s_info
2  {
3    "readers.myreader",
4    "My Awesome Reader",
5    "http://link/to/documentation"
6  };
7
8  CREATE_SHARED_STAGE(MyReader, s_info)

This method will process a options for the reader. In this example, we are setting the z_scale value to a default of 1.0, indicating that the Z values we read should remain as-is. (In our reader, this could be changed if, for example, the Z values in the file represented mm values, and we want to represent them as m in the storage model). addArgs will bind values given for the argument to the m_scale_z variable of the stage.

1  void MyReader::addArgs(ProgramArgs& args)
2  {
3    args.add("z_scale", "Z Scaling", m_scale_z, 1.0);
4  }

This method registers the various dimensions the reader will use. In our case, we are using the X, Y, and Z built-in dimensions, as well as a custom dimension MyData.

1  void MyReader::addDimensions(PointLayoutPtr layout)
2  {
3    layout->registerDim(Dimension::Id::X);
4    layout->registerDim(Dimension::Id::Y);
5    layout->registerDim(Dimension::Id::Z);
6    layout->registerOrAssignDim("MyData", Dimension::Type::Unsigned64);
7  }

This method is called when the Reader is ready for use. It will only be called once, regardless of the number of PointViews that are to be processed.

1  void MyReader::ready(PointTableRef)
2  {
3    m_index = 0;
4    SpatialReference ref("EPSG:4385");
5    setSpatialReference(ref);

This is a helper function, which will convert a string value into the type specified when it’s called. In our example, it will be used to convert strings to doubles when reading from the input stream.

 1  template <typename T>
 2  T convert(const StringList& s, const std::string& name, size_t fieldno)
 3  {
 4      T output;
 5      bool bConverted = Utils::fromString(s[fieldno], output);
 6      if (!bConverted)
 7      {
 8          std::stringstream oss;
 9          oss << "Unable to convert " << name << ", " << s[fieldno] <<
10              ", to double";
11          throw pdal_error(oss.str());
12      }
13
14      return output;

This method is the main processing method for the reader. It takes a pointer to a PointView which we will build as we read from the file. We initialize some variables as well, and then reset the input stream with the filename used for the reader. Note that in other readers, the contents of this method could be very different depending on the format of the file being read, but this should serve as a good start for how to build the PointView object.

1  {
2    PointLayoutPtr layout = view->layout();
3    PointId nextId = view->size();
4    PointId idx = m_index;
5    point_count_t numRead = 0;

In preparation for reading the file, we prepare to skip some header lines. In our case, the header is only a single line.

1    size_t HEADERSIZE(1);
2    size_t skip_lines((std::max)(HEADERSIZE, (size_t)m_index));

Here we begin our main loop. In our example file, the first line is a header, and each line thereafter is a single point. If the file had a different format the method of looping and reading would have to change as appropriate. We make sure we are skipping the header lines here before moving on.

1    size_t line_no(1);
2    for (std::string line; std::getline(*m_stream->stream(), line); line_no++)
3    {
4      if (line_no <= skip_lines)
5      {
6        continue;

Here we take the line we read in the for block header, split it, and make sure that we have the proper number of fields.

 1      // MyReader format:  X::Y::Z::Data
 2      StringList s = Utils::split2(line, ':');
 3
 4      unsigned long u64(0);
 5      if (s.size() != 4)
 6      {
 7        std::stringstream oss;
 8        oss << "Unable to split proper number of fields.  Expected 4, got "
 9            << s.size();
10        throw pdal_error(oss.str());

Here we take the values we read and put them into the PointView object. The X and Y fields are simply converted from the file and put into the respective fields. MyData is done likewise with the custom dimension we defined. The Z value is read, and multiplied by the scale_z option (defaulted to 1.0), before the converted value is put into the field.

When putting the value into the PointView object, we pass in the Dimension that we are assigning it to, the ID of the point (which is incremented in each iteration of the loop), and the dimension value.

 1      std::string name("X");
 2      view->setField(Dimension::Id::X, nextId, convert<double>(s, name, 0));
 3
 4      name = "Y";
 5      view->setField(Dimension::Id::Y, nextId, convert<double>(s, name, 1));
 6
 7      name = "Z";
 8      double z = convert<double>(s, name, 2) * m_scale_z;
 9      view->setField(Dimension::Id::Z, nextId, z);
10
11      name = "MyData";
12      view->setField(layout->findProprietaryDim(name),
13                     nextId,

Finally, we increment the nextId and make a call into the progress callback if we have one with our nextId. After the loop is done, we set the index and number read, and return that value as the number of points read. This could differ in cases where we read multiple streams, but that won’t be covered here.

1      nextId++;
2      if (m_cb)
3        m_cb(*view, nextId);
4    }
5    m_index = nextId;
6    numRead = nextId;

When the read method is finished, the done method is called for any cleanup. In this case, we simply make sure the stream is reset.

1  void MyReader::done(PointTableRef)
2  {
3    m_stream.reset();

Compiling and Usage

The MyReader.cpp code can be compiled. For this example, we’ll use cmake. Here is the CMakeLists.txt file we will use:

 1cmake_minimum_required(VERSION 3.13)
 2project(ReaderTutorial)
 3
 4find_package(PDAL 2.5 REQUIRED CONFIG)
 5
 6set(CMAKE_CXX_STANDARD 17)
 7set(CMAKE_CXX_STANDARD_REQUIRED ON)
 8
 9add_library(pdal_plugin_reader_myreader SHARED MyReader.cpp)
10target_link_libraries(pdal_plugin_reader_myreader PRIVATE ${PDAL_LIBRARIES})
11target_include_directories(pdal_plugin_reader_myreader PRIVATE
12                            ${PDAL_INCLUDE_DIRS})
13target_link_directories(pdal_plugin_reader_myreader PRIVATE ${PDAL_LIBRARY_DIRS})

If this file is in the directory containing MyReader.hpp and MyReader.cpp, simply run cmake ., followed by make. This will generate a file called libpdal_plugin_reader_myreader.dylib.

Put this dylib file into the directory pointed to by PDAL_DRIVER_PATH, and then when you run pdal --drivers, you should see an entry for readers.myreader.

To test the reader, we will put it into a pipeline and output a text file.

Please download the pipeline-myreader.json and test-reader-input.txt files.

In the directory with those two files, run pdal pipeline pipeline-myreader.json. You should have an output file called output.txt, which will have the same data as in the input file, except in a CSV style format, and with the Z values scaled by .001.

Streaming Reader

Streaming points from a cloud can be accomplished via creating a custom writer class that will query the file reader. An example of this, which also shows all the member functions that are needed for a writer, is in examples/reading-streamer.

Fine-grained Streaming Control

Normally PDAL expects that the points will be streamed from a file without any interruption, and be consumed as they arrive. An example showing how to pause/resume streaming points is in examples/batch-streamer.

This example also shows how to use a callback, rather than creating a full writer class. All the variables that must be shared are global.