# Writing a reader¶

Authors: Bradley Chambers, Scott Lewis brad.chambers@gmail.com 11/02/2017

PDAL’s command-line application can be extended through the development of reader functions. In this tutorial, we will give a brief example.

First, we provide a full listing of the reader header.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 // MyReader.hpp #pragma once #include #include #include namespace pdal { class MyReader : public Reader { public: MyReader() : Reader() {}; std::string getName() const; private: std::unique_ptr m_stream; point_count_t m_index; double m_scale_z; virtual void addDimensions(PointLayoutPtr layout); virtual void addArgs(ProgramArgs& args); virtual void ready(PointTableRef table); virtual point_count_t read(PointViewPtr view, point_count_t count); virtual void done(PointTableRef table); }; } 

m_stream is used to process the input, while m_index is used to track the index of the records. m_scale_z is specific to MyReader, and will be described later.

 1 2 3 4 5  virtual void addDimensions(PointLayoutPtr layout); virtual void addArgs(ProgramArgs& args); virtual void ready(PointTableRef table); virtual point_count_t read(PointViewPtr view, point_count_t count); virtual void done(PointTableRef table); 

Various other override methods for the stage. There are a few others that could be overridden, which will not be discussed in this tutorial.

Note

See ./include/pdal/Reader.hpp of the source tree for more methods that a reader can override or implement.

## The source¶

Again, we start with a full listing of the reader source.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 // MyReader.cpp #include "MyReader.hpp" #include namespace pdal { static PluginInfo const s_info { "readers.myreader", "My Awesome Reader", "http://link/to/documentation" }; CREATE_SHARED_STAGE(MyReader, s_info) std::string MyReader::getName() const { return s_info.name; } void MyReader::addArgs(ProgramArgs& args) { args.add("z_scale", "Z Scaling", m_scale_z, 1.0); } void MyReader::addDimensions(PointLayoutPtr layout) { layout->registerDim(Dimension::Id::X); layout->registerDim(Dimension::Id::Y); layout->registerDim(Dimension::Id::Z); layout->registerOrAssignDim("MyData", Dimension::Type::Unsigned64); } void MyReader::ready(PointTableRef) { SpatialReference ref("EPSG:4385"); setSpatialReference(ref); } template T convert(const StringList& s, const std::string& name, size_t fieldno) { T output; bool bConverted = Utils::fromString(s[fieldno], output); if (!bConverted) { std::stringstream oss; oss << "Unable to convert " << name << ", " << s[fieldno] << ", to double"; throw pdal_error(oss.str()); } return output; } point_count_t MyReader::read(PointViewPtr view, point_count_t count) { PointLayoutPtr layout = view->layout(); PointId nextId = view->size(); PointId idx = m_index; point_count_t numRead = 0; m_stream.reset(new ILeStream(m_filename)); size_t HEADERSIZE(1); size_t skip_lines(std::max(HEADERSIZE, (size_t)m_index)); size_t line_no(1); for (std::string line; std::getline(*m_stream->stream(), line); line_no++) { if (line_no <= skip_lines) { continue; } // MyReader format: X::Y::Z::Data StringList s = Utils::split2(line, ':'); unsigned long u64(0); if (s.size() != 4) { std::stringstream oss; oss << "Unable to split proper number of fields. Expected 4, got " << s.size(); throw pdal_error(oss.str()); } std::string name("X"); view->setField(Dimension::Id::X, nextId, convert(s, name, 0)); name = "Y"; view->setField(Dimension::Id::Y, nextId, convert(s, name, 1)); name = "Z"; double z = convert(s, name, 2) * m_scale_z; view->setField(Dimension::Id::Z, nextId, z); name = "MyData"; view->setField(layout->findProprietaryDim(name), nextId, convert(s, name, 3)); nextId++; if (m_cb) m_cb(*view, nextId); } m_index = nextId; numRead = nextId; return numRead; } void MyReader::done(PointTableRef) { m_stream.reset(); } } //namespace pdal 

In your reader implementation, you will use a macro to create the plugin. This macro registers the plugin with the PDAL PluginManager. In this case, we are declaring this as a SHARED stage, meaning that it will be loaded at runtime instead of being linked to the main PDAL installation. The macro is supplied with the class name of the plugin and a PluginInfo object. The PluginInfo objection includes the name of the plugin, a description, and a link to documentation.

 1 2 3 4 5 6 7 8  static PluginInfo const s_info { "readers.myreader", "My Awesome Reader", "http://link/to/documentation" }; CREATE_SHARED_STAGE(MyReader, s_info) 

This method will process a options for the reader. In this example, we are setting the z_scale value to a default of 1.0, indicating that the Z values we read should remain as-is. (In our reader, this could be changed if, for example, the Z values in the file represented mm values, and we want to represent them as m in the storage model). addArgs will bind values given for the argument to the m_scale_z variable of the stage.

 1 2 3 4  void MyReader::addArgs(ProgramArgs& args) { args.add("z_scale", "Z Scaling", m_scale_z, 1.0); } 

This method registers the various dimensions the reader will use. In our case, we are using the X, Y, and Z built-in dimensions, as well as a custom dimension MyData.

 1 2 3 4 5 6 7  void MyReader::addDimensions(PointLayoutPtr layout) { layout->registerDim(Dimension::Id::X); layout->registerDim(Dimension::Id::Y); layout->registerDim(Dimension::Id::Z); layout->registerOrAssignDim("MyData", Dimension::Type::Unsigned64); } 

This method is called when the Reader is ready for use. It will only be called once, regardless of the number of PointViews that are to be processed.

 1 2 3 4 5  void MyReader::ready(PointTableRef) { SpatialReference ref("EPSG:4385"); setSpatialReference(ref); } 

This is a helper function, which will convert a string value into the type specified when it’s called. In our example, it will be used to convert strings to doubles when reading from the input stream.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15  template T convert(const StringList& s, const std::string& name, size_t fieldno) { T output; bool bConverted = Utils::fromString(s[fieldno], output); if (!bConverted) { std::stringstream oss; oss << "Unable to convert " << name << ", " << s[fieldno] << ", to double"; throw pdal_error(oss.str()); } return output; } 

This method is the main processing method for the reader. It takes a pointer to a PointView which we will build as we read from the file. We initialize some variables as well, and then reset the input stream with the filename used for the reader. Note that in other readers, the contents of this method could be very different depending on the format of the file being read, but this should serve as a good start for how to build the PointView object.

 1 2 3 4 5 6  PointLayoutPtr layout = view->layout(); PointId nextId = view->size(); PointId idx = m_index; point_count_t numRead = 0; m_stream.reset(new ILeStream(m_filename)); 

In preparation for reading the file, we prepare to skip some header lines. In our case, the header is only a single line.

 1 2 3  size_t HEADERSIZE(1); size_t skip_lines(std::max(HEADERSIZE, (size_t)m_index)); size_t line_no(1); 

Here we begin our main loop. In our example file, the first line is a header, and each line thereafter is a single point. If the file had a different format the method of looping and reading would have to change as appropriate. We make sure we are skipping the header lines here before moving on.

 1 2 3 4 5 6  for (std::string line; std::getline(*m_stream->stream(), line); line_no++) { if (line_no <= skip_lines) { continue; } 

Here we take the line we read in the for block header, split it, and make sure that we have the proper number of fields.

  1 2 3 4 5 6 7 8 9 10  StringList s = Utils::split2(line, ':'); unsigned long u64(0); if (s.size() != 4) { std::stringstream oss; oss << "Unable to split proper number of fields. Expected 4, got " << s.size(); throw pdal_error(oss.str()); } 

Here we take the values we read and put them into the PointView object. The X and Y fields are simply converted from the file and put into the respective fields. MyData is done likewise with the custom dimension we defined. The Z value is read, and multiplied by the scale_z option (defaulted to 1.0), before the converted value is put into the field.

When putting the value into the PointView object, we pass in the Dimension that we are assigning it to, the ID of the point (which is incremented in each iteration of the loop), and the dimension value.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14  std::string name("X"); view->setField(Dimension::Id::X, nextId, convert(s, name, 0)); name = "Y"; view->setField(Dimension::Id::Y, nextId, convert(s, name, 1)); name = "Z"; double z = convert(s, name, 2) * m_scale_z; view->setField(Dimension::Id::Z, nextId, z); name = "MyData"; view->setField(layout->findProprietaryDim(name), nextId, convert(s, name, 3)); 

Finally, we increment the nextId and make a call into the progress callback if we have one with our nextId. After the loop is done, we set the index and number read, and return that value as the number of points read. This could differ in cases where we read multiple streams, but that won’t be covered here.

 1 2 3 4 5 6 7 8  nextId++; if (m_cb) m_cb(*view, nextId); } m_index = nextId; numRead = nextId; return numRead; 

When the read method is finished, the done method is called for any cleanup. In this case, we simply make sure the stream is reset.

 1 2 3 4  void MyReader::done(PointTableRef) { m_stream.reset(); } 

## Compiling and Usage¶

The MyReader.cpp code can be compiled. For this example, we’ll use cmake. Here is the CMakeLists.txt file we will use:

 1 2 3 4 5 6 7 8 9 cmake_minimum_required(VERSION 2.8.12) project(ReaderTutorial) find_package(PDAL 1.6.0 REQUIRED CONFIG) add_library(pdal_plugin_reader_myreader SHARED MyReader.cpp) target_link_libraries(pdal_plugin_reader_myreader PRIVATE ${PDAL_LIBRARIES}) target_include_directories(pdal_plugin_reader_myreader PRIVATE${PDAL_INCLUDE_DIRS}) 

If this file is in the directory containing MyReader.hpp and MyReader.cpp, simply run cmake ., followed by make. This will generate a file called libpdal_plugin_reader_myreader.dylib.

Put this dylib file into the directory pointed to by PDAL_DRIVER_PATH, and then when you run pdal --drivers, you should see an entry for readers.myreader.

To test the reader, we will put it into a pipeline and output a text file.

In the directory with those two files, run pdal pipeline pipeline-myreader.json. You should have an output file called output.txt, which will have the same data as in the input file, except in a CSV style format, and with the Z values scaled by .001.