The study of mean curvature flow not only is fundamental in geometry, topology and analysis, but also has important applications in applied mathematics, for instance, image processing. One of the most important problems in mean curvature flow is to understand the possible singularities of the flow and self-shrinkers, i.e., self-shrinking solutions of the flow, provide the singularity models. In this talk, I will describe the rigidity of asymptotic structures of self-shrinkers. First, I show the uniqueness of properly embedded self-shrinkers asymptotic to any given regular cone. Next, I give a partial affirmative answer to a conjecture of Ilmanen under an infinite order asymptotic assumption, which asserts that the only two-dimensional properly embedded self-shrinker asymptotic to a cylinder along some end is itself the cylinder. The feature of our results is that no completeness of self-shrinkers is required. The key ingredients in the proof are a novel reduction of unique continuation for elliptic operators to backwards uniqueness for parabolic operators and the Carleman type techniques. If time permits, I will discuss some applications of our approach to shrinking solitons of Ricci flow.