It is now unclear what impact the judgement will have on the current five week suspension of Parliament, a process known as proroguing, which started in the early hours of Tuesday.
Judge Lord Doherty dismissed the challenge at the Court of Session last Wednesday, saying it was for politicians and not the courts to decide.
Three judges accused the Prime Minister of suspending the Commons for the "improper objective of stymying Parliament" over his Brexit plans and said last Monday's prorogation was "null and of no effect".
Boris Johnson suffered another setback today as Scottish judges ruled his suspension of Parliament is unlawful.
Brexit news: Scotland court rules Boris Johnson's suspension of British Parliament illegal
The decision by a three-judge panel on the Court of Session will not immediately force the government to allow lawmakers to reconvene - their suspension officially began on Tuesday, and they're not slated to meet again until October 14.
The British government said it was disappointed by the decision and confirmed it would appeal to the Supreme Court.
Its ruling was at odds with the High Court in England which said that the shutdown was not illegal even if it was motivated by the desire for "political advantage".
"Huge thanks to all our supporters & our fantastic legal team who have achieved the historic ruling that #prorogation is #unlawful", Scottish National Party lawmaker Joanna Cherry said on Twitter.
"Although Johnson claimed he was shutting down Parliament for five weeks to work on his legislative agenda", NPR's Frank Langfitt reports from London, "most political observers think he was trying to sideline legislators so they couldn't blockhis plan to crash the United Kingdom out of the European Union without a withdrawal agreement".More news: England drops Roy for final Ashes Test
In a sensational move, three judges sitting in Scotland's highest court of appeal found in favour of a group of more than 70 parliamentarians who challenged the Conservative leader's controversial decision to prorogue - or suspend - Parliament for five weeks, which came into effect on Tuesday. But if Boris Johnson hadn't suspended Parliament, would he have had a better chance of getting a general election approved?
A lawyer involved in bringing the case in Scotland suggested it may be heard as soon as next Tuesday.
Sir Keir Starmer, the shadow Brexit Secretary, called on MPs to return to Westminster so they can "open those doors" and hold Mr Johnson to account.
If Boris Johnson refused to do so, the Queen would be plunged into an incredibly awkward position.
"Court says prorogation was unlawful and null and void - so Parliament must be recalled immediately to allow the essential work of scrutiny to continue".More news: Team USA basketball vs. France, score
"I have never been able to contemplate the possibility that the law could be that our sovereign Parliament might be treated as an inconvenience by the prime minister", he said.
Yes, but this would have to be agreed by MPs who vote on whether or not to go into recess.
She added: "We feel utterly vindicated and I would be confident that the UK Supreme Court will uphold this decision".
A full hearing is scheduled at the Supreme Court on September 17.More news: Apple tweaks App Store algorithm to handicap its own apps