UAV-based monitoring system for mites in greenhouse cultivation

TP5 - Karlsruhe University of Applied Sciences (HKA)

Dynamic UAS control and 3D georeferencing

I. Introduction and project concept

MiteSens – Concept, goals and R&D consortium

The aim of the joint project MITESENS, financed by "Bundesanstalt für Landwirtschaft und Ernährung (BLE)", is to develop a UAS (Unmanned Aerial Vehicle) system for monitoring spider mites in typical plant populations (strawberries, cucumbers, roses) in greenhouse cultivation (MiteSens).

Fig.1: MitesSens overall system with tasks and services of the consortium members as well as interactions and data interfaces. (Graphic from Hortico 4.0 Meeting Dec. 9, 2020)

Fig. 1 shows the MitesSens overall concept and the tasks of the various members (University of Hohenheim, University of Applied Sciences Karlsruhe (HKA), Wolution, Multikopter, IB Bauer, LVG Heidelberg and LTZ Augustenberg) as well as the interactions and the data interfaces. The project duration is March 31, 2023.

MiteSens should be able to detect the early infestation of plant leaves with spider mites, as well as to monitor appropriate control measures with predators and / or integrable pesticides (acaricides) and to evaluate their success. MiteSens is based on imaging processes. In connection with a UAS (MiteSens UAS) with intelligent flight control as a camera carrier, a high spatiotemporal resolution of the spider mite monitoring, a contactless use and an autonomous application are possible.

The derivation of the necessary infestation information (location, strength, spatiotemporal dynamics of the spider mite infestation) from the located image information is based on an ML (machine learning) approach, which carries out the corresponding evaluations in real time, thus enabling a quick reaction to a possible infestation.

The central component of the overall MiteSens system is a UAS (Unmanned Aerial System) that performs several tasks in connection with a ground control station for data communication both with the UAS and with the cloud as a general technical data interface of the consortium (Fig. 1). On the UAV (Unmanned Aerial Vehicle) as the sensor and communication carrier platform of the UAS - in addition to the multispectral camera used to monitor the mite infestation - the GNSS, MEMS (Micro-Electro-Mechanical-Systems) sensors and optical sensors are located in compact form (Multispectral camera, optical cameras for navigation and obstacle detection, laser scanner (LIDAR) for 3D inventory as a voxel model) and finally the data communication components (WIFI, Internet).

The GNSS / MEMS / optics multi-sensor system is primarily used for navigation and the associated flight control, i.e.

• Determination of the 18 parametric navigation status y(t) (Fig. 2).
• Dynamic obstacle detection using optical sensors (camera, laser scanner)
• Automated flight control (FC) along a 3D trajectory y(t)_target

The actual navigation status y(t)_Ist can be used in conjunction with the laser scanner, which is used for the 3D building and inventory recording, in a dual function for the navigation status estimation. In the case of a state estimation extended by the model m (t), one speaks of SLAM (Simultaneous Localization and Mapping) (y (t)_Ist, m (t)).

Basically, the ETRF89 actual navigation status y (t) is determined in the global ITRF (International Terrestrial Reference Frame) or in Europe in conformity with INSPIRE as a spatial reference, i.e. globally identifiable in terms of location and time

• 3D georeferencing of the laser point clouds m (t) and the existing buildings and plants obtained from them as well as the unclassified and classified multispectral images s (t) and s (t) ‘.
• Based on the point clouds BIM-compliant products will be created
  
  • 3D building model b and
  • 3D voxel model v of the plant population

Using image processing methods, the 3D building model b as a digital twin of the building flown through can then in turn be used as a marker for the MEMS / optics data fusion in indoor navigation (has to do without the GNSS component). The classified multispectral images s (t) ‘are then also calculated back to the 3D voxel model v on the basis of the navigation state vector y (t) actual. This finally results in the ITRF / ETRF89 georeferenced spatial 3D voxel model v ‘of the mite population. Vertical or horizontal sections (2D ortho maps) can then be derived from the complete 3D information v ‘.

Fig. 2: Sensors of the MiteSens UAS Flight-Control (FC) for the state estimation y (t), spider mite detection and data communication

Further project information can also be found on the website of the funding agency (Bundesanstalt für Landwirtschaft und Ernährung) under the link below https://service.ble.de/ptdb/index2.php?detail_id=18370053&site_key=141&sLfd=laufend&zeilenzahl_zaehler=1684&NextRow=1550

II. Dynamic 3D flight control of the UAS

A first major challenge for the autonomous indoor flight of a UAS as a carrier platform for applications - all of which can be georeferenced via the navigation status vector - is to develop the associated flight control (FC) as an overall system in hardware, algorithms and software.

The autonomous controls and corresponding control algorithms of land, sea and air vehicles are generally based on the so-called control difference, only the actuators and the relevant manipulated variables are different. The regulation is understood to mean the continuous comparison of “actual versus target” with the premise of establishing identity over a period of regulation. The flight physics of UAS is based on the time-dependent 3D total thrust vector f (t) generated from the sum of the individual propeller speeds and the 3D total torque vector based on the Euler and Newton equations. From these, with the mass m and the inertia matrix J of the UAS, the 6 translational and rotational speed terms and the 6 translational and rotational acceleration terms can be determined.

For automated flight control along a spatial trajectory - in contrast to the above-mentioned remote control or the joystick - the target state y (t)_target or the control deviation e (t) are now defined exclusively via the trajectory profile via assigned properties. In the first instance, these are the sequence of 3D spatial points P_{(i, Target)} to be flown through, which define the 3D trajectory and the speed profile linked to the 3D trajectory (speed amount along the trajectory). This results in a control deviation vector that contains the components of the position deviation in the transverse component, the speed deviation in the transverse component and the speed deviation in the longitudinal component of the trajectory.

The control deviations lead to a PID controller for the two lateral deviation components and a PI controller for the longitudinal deviation in autonomous trajectory flight.

Fig. 3: Definition of trajectories via the 3D polygon of desired waypoints P_i,target

III. Georeferencing and Photogrammetry

The overall objective of the MiteSens project is the development of a UAV-based monitoring system for spider mites (Tetranychidae) infestation in greenhouse plantations of Strawberries, Cucumbers and Roses.The monitoring system depends on machine learning based classification of data cubes (images) captured bya hyper-spectral camera mounted on a UAV.

The critical information that need to be extracted from the images include location, strength and space-timedynamics of the infestation. In order to acquire spatial information (location) of the detected infestation,image pixels should be related to spatial coordinates which can either be local or global (georeferencing) viaphotogrammetric process.

In general photogrammetry is required to provide 3D real world coordinates of objects derived from image measurements so that further elements such as distances, areas and volumes can be computed. The following are specific objectives for photogrammetric and georeferencing tasks in MiteSens project :

• To provide accurate spatial reference (location) for interest objects (spider mites)
• To provide an orthomosaic map of the region of interest (ROI) which can serve as a base map to displayother spatial information on top of
• To provide height information of plant growth
• To provide visualization of symbolic (classified) images as an overlay on top of an orthomosaic and as a 3-dimensional classified point cloud (voxel) map

In addition, extended photogrammetric products such as point clouds and voxel models will be used internally for autonomous navigation and obstacle avoidance. Such models become inevitable specially in cases where the operation environment changes.

Point cloud georeferencing

A typical georeferencing workflow involves the use of a ground-truth trajectory y(t), i.e. coming from the navigation vector determination, and the LiDAR pointcloud timestamps to get the position and orientation for points in the point cloud. Fig.4 shows the packetbased timestamping of a typical multi-beam 3D LiDAR sensor. Although revolution based timestamping suffices for slowly moving platforms (under 1 m/s), a complete packet timestamp based georeferencing is implemented in MiteSens project.

Figure 4: Top-down view of point clouds of a single revolution colorized with unique packet timestamps

Georeferencing is done as per the standard LiDAR georeferencing equation given as

where:

• P_G^e coordinates of the target point in the global reference frame
• P_b^e coordinates of the body in the global reference frame
• R_b^e Rotation matrix of attitude (Roll, Pitch, Yaw) of the body in global frame
• R_s,i,j^b Rotation matrix from LIDAR frame to the navigation frame (boresight calibration matrix)
• r^s coordinates of a single point in the LIDAR frame
• l_i,j^b coordinates of the LIDAR (sensor index i, platform index j) in body frame (lever arm)

Figure 5a: Georeferenced point cloud visualised on top of a satellite image.

Figure 5b: 3D Point cloud perspective view.

Figure 5: Preliminary georeferencing results based on open source dataset NCLT (Carlevaris-Bianco et al.,2016).

Image Georeferencing

Surface models and point clouds can be constructed using a local coordinate system. To relate image data to a global coordinate system, a process of converting the local coordinate system into a world coordinate system (georeferencing) must be applied. Location information in MiteSens project could be available either from a high-accuracy RTK based navigation estimation provided by the flight control (navigation) or through the use of pre-surveyed ground control points (markers).

In the presence of a reliable and accurate position and orientation data, a datum for the object coordinate frame can be defined by direct measurement of the exterior orientations without using any reference points, a procedure known as direct georeferencing. Alternatively, an integrated georeferencing approach where direct sensor data is combined with reference points such as via optical markers shall be used.

The output of a georeferencing procedure is a transformation matrix which allows the conversion of pointclouds and 3D models from local to global coordinates. The figure below gives an overview of the transformations involved in the georeferencing procedure.

Figure 6: Concept of georeferencing image points

IV. FC-Hardware

[in progress]

V. References

Carlevaris-Bianco, N., Ushani, A. K., & Eustice, R. M. (2016, August). University of Michigan North Campuslong-term vision and lidar dataset.The International Journal of Robotics Research,35(9), 1023–1035. doi:10.1177/0278364915614638