Learning to Detect and Predict Contact Events on Vision-based Tactile Sensor

We propose to classify and predict tactile signal using deep learning method, seeking to enhance the adaptability of the robotic grasp system to external disturbances. We develop a deep learning framework and collect tactile image sequence with vision-based tactile sensor, FingerVision. The neural network is integrated into a FingerVision-based robotic grasping system to detect the current and predict the future grasping state, e.g. rolling, slipping or stable contact.

Active Perception with A Monocular Camera for Multiscopic Vision

Utilizing a low-cost monocular RGB camera to acquire accurate 3D information is crucial to many robotic applications. We propose an active perception framework to create a super stereo structure, which we refer to as multiscopic vision. In this system, the camera positions are flat co-planar and with the same horizontal or vertical parallax. Both the magnitude and direction of the pixels disparities is under control such that we can search the correspondence easily. With this strong regularization, we fuse the cost volumes from different images, which makes the correspondence searching more robust to outliers. The obtained disparity maps are more accurate, especially in occluded areas, reflective regions and feature-less noisy parts.

Sorting Cluttered Tabletop Using Improved Monte Carlo Tree Search

In industrial environments, there are many tasks about sorting and rearranging different kinds of objects. Inspired by AlphaGo, we propose an improved MCTS to do this. We use the results of a classical MCTS to train a policy network and then use the network to guide the tree search.

Manipulation Using Deep Reinforcement Learning in Topology Space

We model a whole arm manipulation task about holding as a deep reinforcement learning problem in order to provide a behavior that can directly respond to external perturbation and target motion. To improve the performance of deep learning in robotics application, we propose a new state as the input of the networks, the topology representation. This state allows transferring the learned policy to various shapes, sizes and poses because they are the same in topology space. Compared to RGB image state or pose coordinates state, it can better describe the interaction state between the robot and environments. Besides, there is no reality gap between simulation and reality, which means the policy trained in simulator can be directly transferred to real world.

Rearrangement Using Deep Reinforcement Learning and Transfer Learning

In contrast to traditional methods which can only handle static scene, as explicitly modeling the physical environment is not always feasible and involves various uncertainties, we learn a nonprehensile rearrangement policy with deep reinforcement learning in simulation and then transfer the knowledge to reality. We propose potential-field-based heuristic exploration and reply buffer curation to assist the training and propose a transfer method in Q-space to fill in the gap between simulation and reality. The real robot can perform well in the dynamic scene with our approach.

Machine Vision Research and Software Development of Intelligent Forging Production Line Based on Industrial Robots (Graduation Project)

Set up a special hardware platform to acquire usable images. 
Designed a system for high temperature forging which can obtain its the size, position and posture. 
Developed an interaction interface based on MFC to output the detection information and give the alarm when the position or posture of the forging is wrong.

On-Road Lane Detection System Based on Machine Vision

Detect the structured and unstructured road edge based on the gradient of brightness and the RGB information of the road to assist the driving of a Fuel Cells Go-Kart.

Ninth Robot Competition of Zhejiang Province (Team leader)

  • Wrote the program completing two tasks (4 tasks in total in the robot soccer game): goalie task and finding the best shooting position.
  • Participated in designing the algorithm of path planning task and positioning task and got the second prize.