Android CoordinatorLayout with keyboard

I am trying to use CoordinatorLayout

to bind a view to another and it works almost perfectly. The problem is that when I switch to another snippet, open the keyboard and then go back, the anchored view gets confused. Sometimes after a couple of seconds, it will (apparently) randomly snap into place. Here's a gif to demonstrate what I'm talking about.

As you can see, it FloatingActionButton

doesn't align when I go back to the fragment, but after a while it snaps into place again. What could be causing this problem?

XML snippet

<?xml version="1.0" encoding="utf-8"?>
<android.support.design.widget.CoordinatorLayout android:id="@+id/coordinatorLayout_main"
                                             xmlns:android="http://schemas.android.com/apk/res/android"
                                             xmlns:app="http://schemas.android.com/apk/res-auto"
                                             android:layout_width="match_parent"
                                             android:layout_height="match_parent"
                                             android:fitsSystemWindows="false">

    <!-- Other views -->

    <!-- Bottom sheet -->
    <include
        android:id="@+id/bottom_sheet"
        layout="@layout/bottom_sheet_primes"/>

    <android.support.design.widget.FloatingActionButton
        android:id="@+id/button_start"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginEnd="@dimen/fab_margin"
        android:src="@drawable/ic_play_arrow_white_24dp"
        app:backgroundTint="@color/accent"
        app:layout_anchor="@id/bottom_sheet"
        app:layout_anchorGravity="top|end"/>

</android.support.design.widget.CoordinatorLayout>