Prophet's Tribe, Ahlal Bayt (Family) & Righteous Successors
The eminence of Banu Hashim and their descendants Banu Abbas
In order to understand why Banu Abbas, descendants of Prophet’s uncle Abbas (r.a) are considered Prophet Muhammad’s family and no other Arab tribes of the Quraish including Banu Ummayad, even though they all share common ancestry with the Prophet ﷺ, one has to realize the status of the Prophet ﷺ , his family, his tribe and his ancestors as opposed to the rest of the humankind. A prophetic hadith teaches us: “Verily Allah granted eminence to Kinana from amongst the descendants of Ismail, and He granted eminence to the Quraish amongst Kinana, and He granted eminence to Banu Hashim amonsgst the Quraish, and He granted me eminence from the tribe of Banu Hashim.” These prophetic words emphasize special honor granted to Kinanah, an Adnani descendant from line of Ismail (alaihi salaam), and then to Quraysh from the line of Kinanah. Among the descendants of Quraysh was a man named Abd Manaf. He had four sons: Abd Shams, Nawfal, Muttalib and Hashim. Allah ﷻ selected Prophet Muhammad ﷺ from the descendants of Hashim and therefore Banu Hashim has excelled all others descendants of Abd Manaf and excelled all other Arabs and the rest of humankind in gaining special honor. Hashim had a son named Abdul Mutallib (Prophet’s paternal grandfather) and one of his sons is Abbas ibn Abdul Muttalib (r.a), ancestor of Banu Abbas and Prophet’s uncle. Prophet’s father Abdullah (r.a) was also one of the sons of Abdul Mutallib which is why Prophet’s name is Muhammad bin Abdullah bin Abdul Mutallib.
As for Banu Ummaya, they are descended from one of Abd Manaf’s son Abd Shams. As per Prophet, divine decree did not choose select the descendants of Abd Shams for prophet’s ancestry, therefore they are not considered having special honor and part of the prophet’s Ahlal Bayt (family). Banu Hashim includes the children of Prophet’s paternal uncle Abbas (r.a), children of Ali (r.a) and Ali’s brothers such as Aqil (r.a) and his children, and Jafar (r.a) and his children. They are all considered as Prophet’s household, his Ahlal Bayt (family), as per view of the majority Muslims who belong to school of Ahle Sunnah Wa’al Jamaah (People of the way and consensus). Since Abbas (r.a) children are considered ahlal bayt therefore when they replaced Banu Ummaya as the rulers of the Muslims (Abbasid Dynasty) they were considered as bringing the rulership back to the Prophet’s family.
 Kinana, an ancestor of Quraysh tribes was an Adnanite Arab, descendants of Ismail (alaihi salaam) through his son Adnan.
 Sahih Muslim 2276 – Sunnah.com