By Chris Brown

In a lot of ways the agriculture industry led the way with the widespread application of autonomous vehicle technology. Autono-
mous Solutions Inc. (ASI) started working with John Deere in U.S. in 2000. Over the next 10 years, ASI automated every vehicle in Deere’s fleet for a variety of agriculture processes. The company then branched into applications for military R&D and the automotive industry. In 2006, ASI entered the mining business working with Phelps Dodge (now Freeport McMoRan Copper & Gold), automating a haul truck and a dozer. The company is now working with Barrick, Rio Tinto and several other mining companies to develop autonomous technology for mining applications.

In a relatively short period of time, ASI has automated 50 different kinds of vehicles—not 50 vehicles, 50 different kinds of vehicles. The company has a good understanding of what it takes to automate a vehicle and integrate it into a process. Most of the work involves surface applications, using GPS-based positioning for navigation. The company has some experience with factory and underground applications, or what is referred to as GPS-denied positioning. That knowledge is helpful in addressing deficiencies with GPS.

There are two approaches to introducing autonomous technology in the mine environment, an all-or-nothing grand vision approach, where the mine goes from manned-vehicles to a fully-automated unmanned fleet in one step, or a moderated user-assisted approach. The user-assisted approach has been implemented fairly successfully in the agriculture industry for the last several years. Before a mine makes that decision though, understanding autonomous vehicles, how they work and robotics, sheds light on the technology’s strengths and weaknesses.

Who is looking at robotics? The interest seems to fall into two camps, which are not mutually exclusive, but the interest tends to be dominated by one or the other. Many major mining companies are considering autonomous mining to improve efficiency and productivity. They have a long range vision and they are expecting to make an investment to significantly change the mining process. These mining companies may also have “excessive manpower problems,” or they might operate in remote or unpleasant locations or both.
Then, there are also those mine operators with immediate, non-routine safety problems, such as geotechnical issues. A highwall or bench failure would be a good example, where the mine has been told by regulators or an insurer that they cannot operate in a certain area and they need an unmanned vehicle to perform the work. These companies have safety concerns and are not necessarily looking to make an investment. They are not expecting to improve productivity. If anything, they are expecting to take a hit on productivity by using a remote control machine to deal with a circumstance.

Automation can be used to improve the mining process. The technology could allow equipment to work through breaks/shift changes, to mitigate the effects of weather or fatigue, and to eliminate mistakes by improving the accuracy and repeatability. Accurately repeating processes is an important aspect for miners. Haul trucks could be spotted correctly every time (dumping and loading). They would travel at the right speed all of the time. They would place materials in the appropriate place consistently. The machines would move/rip as much rock as possible. The system could maximize hole drilling speed and accuracy.

By being able to improve all of the pieces in the process, ultimately with the grand vision of a fully autonomous mine, engineers begin to redesign the mine and take a whole new approach.

The all-or-nothing approach is the approach currently being embraced by the OEMs. It is a viable option that requires a significant investment in one single integrated solution from one OEM. The potential upside is obvious. This approach will work best for new mines in remote locations. Aside from the big investment, the downside would be the reliability risks associated with a big process change and working with a single supplier.

Miners could also opt for an incremental approach. They could begin with a user-assisted approach before transitioning to full autonomy over time. John Deere had an all-or-nothing vision. They were going to go from where they were to unmanned farming in one step. That hasn’t happened in the last 10 years. Instead, they have taken an incremental approach where they introduced user-assisted systems. Year-by-year, the user-assisted systems took more control of the functions of the vehicles until ultimately farmers have unmanned agriculture.

What is an Autonomous Vehicle?
To truly understand the grand approach as well as the incremental approach, users must first get a feel for how the technology works. Most people have some hazy notion that it has something to do with GPS. As the technology on an unmanned vehicle matures to an autonomous state with see-and-avoid behavior, it progresses through several stages. It starts with remote control, which electronically controls all the degrees of freedom on the vehicle. The operators have a controller in their hands and they are looking at the vehicle. The next level is tele-remote operation, the distinction is that the operators can’t see the vehicle so they are dependent on a video feed. For the next level, autonomous (blind), GPS and mapping software give the machine the ability to execute a sequence of paths and actions commanded by the user, but it’s blind. If there is an obstacle in the way, the vehicle will plow into it or over it in the case of a haul truck. This would be the lowest level of autonomy. Vehicles can perceive their environment using an array of sensors. The sophistication varies from see-and-stop behavior, where a vehicle has a sensor horizon that tells it there is something there, but not enough sensors to tell it how to safely plan around it, to see-and-avoid behavior.

The first step in automating a vehicle is to convert it to drive-by-wire by adding actuators and hydraulic controls. Once it is drive-by-wire enabled, computers can talk to it. Most vehicles are not equipped with drive-by-wire controls. The drive-by-wire system ties into a controller, which acts as the brain for the vehicle. The controller is getting data from position sensors, which are primarily GPS-based, telling the vehicle where it is located. It also receives data from obstacle detection (OD) sensors. It knows where it is and what’s around it. The operator uses that information to control the vehicle, communicating with the base station command-and-control software over a radio network. The video system operates in parallel.

The most vehicle-specific aspect of automating the vehicle is the drive-by–wire system. When ASI is approached with a new vehicle, the engineers determine what devices will be used to control the vehicle. Everything else—all of the other black boxes—remain the same from vehicle to vehicle. To a robotics engineer involved in autonomous systems, a utility vehicle is no different than a haul truck even though the vehicles are worlds apart as far as size and weight. Typically, the drive-by-wire package controls the steering, brakes, throttle, transmission, attachments and ancillary functions (horn, lights, etc.). It receives data from OEM systems (speeds, engine RPM, health, etc.) The goal in designing a good drive-by-wire system is to minimize the impact on the base platform and always allow a human to easily take control and operate the vehicle.

The division of labor between the command-and-control software and the vehicle controller is straightforward. On the command-and-control side, the software receives high level goals from the user, e.g., go from point A to point B and do something. It’s creating a global path plan to  execute the goal taking into account the map and the vehicle’s driving capabilities. It sends the plan either all at once or in chunks at a time over the radio network to a vehicle control unit (VCU). On the VCU locally, the controller takes the goal and does all of the low level computing to determine how it has to manipulate the throttle, steering and transmission to execute the goal. The VCU closes control loops on the vehicle. It performs onboard safety checks for off-path error, over-speed, over RPM, steering limits, redundant sensing, etc.
GPS is not perfect and it requires filtering and augmentation, especially in robust environments such as mining. It fits hand-in-hand with good map data. Adding perception and intelligence to the system mitigates the need for high accuracy GPS and/or map data. An inertial measurement device, which calculates with a six-axis set of accelerometers and gyros, tells the system how the vehicle is moving. With vehicle odometry, the system knows the speed of the vehicle and the angle of the steering wheel. If it loses GPS, the system can dead reckon where the vehicle should be until it gets a new GPS point. A variety of other sensors perform tasks of similar complexity.

For GPS-denied situations, infrastructure-based solutions, such as reflectors and RFID tags, can be positioned throughout the pit. Another technology currently under development is vision- or laser-based recognition, where the system recognizes landmarks. The military is interested in this technology because of the threat that GPS could go away. That is the direction the industry will move toward for unmanned robotic systems.

Obstacle Detection and Avoidance
ASI uses three types of sensors for obstacle detection: lasers, vision and radar. They all have strengths and weaknesses. What stays the same is the obstacle detection processing software, which takes the data from whatever suite of OD sensors make sense for a given application and converts that into a traversable map for the vehicle, locations it can and cannot drive. The specific selection of sensors depends on a variety of factors based on definition. Would the vehicle be looking for other vehicles, people or debris on the road? The vehicle’s speed and the look-ahead distance would also be a consideration for safety purposes. Environmental factors, such as rain, snow, dust and fog are also considerations. The sensors are fairly expensive, so mining companies would want to use as few of them as possible on each vehicle.

On the command-and-control side, the software should perform multi-vehicle coordination. The software has to be able to take high-level goals from a dispatch system and coordinate the activities of multiple vehicles to achieve those goals. The best situation avoids deadlocks or stalemates along the route. Instead, the software prioritizes the vehicles and prevents them from ever coming near each other.
A whole new set of safety issues arise from autonomous mining. Does the mine allow humans and manned vehicles to interact with unmanned vehicles? How willing is the mine to rely on the OD system or tagging process to govern safety? One option is to use GPS-based safety perimeters and “keep out” zones. That would probably be impractical for haul trucks that traverse the entire pit on haul roads shared with other vehicles. There are also a couple layers of safety measures built into the system, while it is working. As previously discussed, long-range path planning for multiple vehicles prevents them from encountering each other. If they do encounter another, then the system moves from a centralized approach to a decentralized peer-to-peer approach where the vehicles can ultimately see each other. Even if they are not communicating with each other or they are not updating their position to a central controller, they can still see each other.

When the system isn’t working properly for some reason, it needs to fail in a safe manner. There are a lot of low level features that assist with the fail-safe system. An autonomous system must have a software-independent, stored-energy emergency brake system. In the case of complete power loss, it will bring the vehicle to a complete stop. There are several triggers, such as communications loss, off-path error, and system failure, which include redundant sensing on vehicle controls.

If radio communications is lost, then the big red button in the control room no longer works. Theoretically, the vehicle can run without radio communications. It could get the entire mission from the global controller and then off it goes. It does not need to talk to the global controller again. That’s probably not the safest way to operate. Typically, there is some kind of time-out on the vehicle if it loses communications.
There are many questions surrounding autonomous mining. At what point will the technology be proven? There is a big difference between seeing a demo and seeing a system run 24-7, 365 days a year. How would a mine recover from failure if it has no operators? Reliability is obviously a huge issue. How does the mine know an OD system is safe enough? What type of test(s) does it run? There is no industry standard at this point. Can the system be integrated into existing operations?

Incremental Options
Working with a user-assisted program, miners could move toward the big vision of autonomous mining incrementally over time. The 24-7, 365 performance goal would be tabled for a while, but the mine could implement the technology at existing operations and retrofit the technology to current fleets. Humans are still in the loop and can be safely integrated into manned operations. Some of the current user-assisted systems for mining include auto-spot and auto-pilot solutions for haul trucks, collision alert-avoidance systems, remote control/tele-remote systems with autonomous assist and ripping path assist for dozers.

Many machines, such as drills, shovels and dozers, have GPS and mapping tools to assist the operator that are also tools that could be used for autonomous operations. The next move would be to shift up to a drive-by-wire or steer-by-wire technology. The users would have the ability to implement auto-spotting or auto-pilot. When the truck driver gets close to the shovel and they are able to communicate with each other, the autonomous system takes over and performs the spotting. Similarly, when the driver enters a long-haul corridor, the auto-pilot takes over and maintains optimal speed. Data from mine operators indicates that average speed tends to slowly decrease over the course of a shift and they could theoretically gain a measurable payback for that investment.

One of the problems with automating a haul truck fleet is the number of vehicles, which is a big investment. They cannot be effectively isolated. There are sets of equipment that do operate in isolated zones, such as drills and dozers. The mines could start by automating multiple unmanned vehicles, which do not have to interact with manned vehicles. In certain situations, such as multiple drills working one pattern or multiple dozers ripping a leach pad, a single operator could supervise two to five vehicles. Eventually as the group (operators, production managers and engineers) become more comfortable with the technology and equipment, it can be phased in over time.

Brown has been involved in the development of robotic systems for military, mining, industrial and agricultural applications for more than 10 years. He is currently vice president of business development for Autonomous Solutions, a Utah-based robotics company focused primarily on converting off-the-shelf vehicles over to autonomous operation. This article was adapted from a presentation that Brown made at E&MJ’s Haulage and Loading conference during May 2011.

| More...